feat(evals): pointcloud VQA suite + PointCloud2.agent_encode (evo autoresearch, 0.136 -> 0.96)

[spomichter]

Problem

we had no way to measure whether an agent can actually read pointcloud geometry — and it turns out it couldn't: str(PointCloud2) hands the model num_points and nothing else. baseline on the new pointcloud VQA evals: 0.136, identical to the blind-guessing control (0.138). the encoding carried literally zero information.

Solution

two things, stacked on #3411 (uses its evals framework as the harness):

1. pointcloud VQA benchmark (evals_bench/): 40 generated cases over go2_short + go2_bigoffice (the long replay). ground truth computed analytically from full-res clouds + odom (privileged modality). families: extent / zspan / area (world geometry), nearest-obstacle clearance from the robot's own pose (metric distance), shift / compass / areatrend over cloud sequences (spatiotemporal). anti-gaming: blind-ablation gate (context withheld must stay ≤0.35), answer-leak grep, target-only diff gate, PointCloud2 unit tests.

2. PointCloud2.agent_encode() — found by evo autoresearch overnight (14 experiments, every commit on this branch is one experiment with its hypothesis + measured effect in the message). final: 0.962 (sighted) vs 0.15 blind. 2.5-D design: exact full-cloud scalars (extent/z-span/footprint) + per-frame world centroid + body-height slab as per-row occupied x-intervals in exact world meters with y-extents, plus closed-form reasoning recipes (clearance dx = max(0, min-qx, qx-max), trig-free 8-way compass).

per-family: extent/zspan/area/shift/direction/areatrend ~1.0, nearest 0.81→0.85 (band-limited, sampling-noisy).

things we learned that generalize (kept in the commit messages): models short-circuit branchy recipes but can't short-circuit closed-form formulas; give decision procedures not trig; every added signal needs a routing note or it becomes a distractor for other question types; exact coordinates beat quantized grids for LLM metric reasoning.

report w/ rendered figures: https://dimos-pc-encoding-report.vercel.app

Breaking Changes

None (new method + new benchmark dir only; diff outside those is empty by gate).

How to Test

uv run pytest dimos/msgs/sensor_msgs/test_PointCloud2.py -q
# full eval run (needs OPENAI_API_KEY + lfs data):
env PYTHONPATH=. uv run python evals_bench/benchmark.py
# guessing ablation:
env PYTHONPATH=. uv run python evals_bench/benchmark.py --blind --max-mean 0.35

Contributor License Agreement

• I have read and approved the CLA

[spomichter]

evo:ship mergeability report (advisory)

Technique. Adds one method pair to PointCloud2 (agent_encode() + _body_height_occupancy()): exact full-cloud scalars (bbox ranges, extent, z-span, 0.2 m-cell floor footprint), per-frame world centroid, and the body-height slab (z 0.15–1.0 m) as per-row occupied x-intervals in exact world meters with per-interval y-extents — plus closed-form reasoning recipes (clearance dx = max(0, min−qx, qx−max) per axis; trig-free 8-way compass rule). ~2.7 kB per observation.

Behavioural correctness. Pointcloud VQA evals: 0.136 baseline → 0.962 shipped (blind control 0.15, gate ≤0.35). Distilled file is byte-identical to winner exp_0013's, so the measured score carries exactly.

Regression safety. test_PointCloud2.py: 3 passed, 1 deselected (pre-existing marker). No other files touched.

Scope. One file, +93 lines, purely additive (new methods; no existing code modified). A git gate enforced target-file-only diffs throughout the search.

Test correctness. No test modified, weakened, or removed at any point (gate-enforced). New behaviour is covered by the eval suite in this PR's evals_bench/ rather than unit tests — worth a unit test for agent_encode() output shape before merge if reviewers want offline coverage.

Mechanical cleanliness. ruff check + format clean; full pre-commit suite passed.

Codebase adherence. Follows the existing points_f32()/bounding_box_dimensions() method style on the class; numpy-only, no new deps.

Branches. This PR carries the full 8-experiment research history. A distilled single-commit equivalent (identical final file, no evals_bench/) is at feat/pointcloud-agent-encode-min (40f22ba7e) if a squashed landing is preferred.

Caveat: exp_0013 (0.962) vs exp_0012 (0.957) is within measured sampling noise on the nearest family — treat as tied; the tip includes both refinements.

[codecov]

Codecov Report

❌ Patch coverage is 85.10638% with 7 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/msgs/sensor_msgs/PointCloud2.py	85.10%	3 Missing and 4 partials ⚠️

@@                  Coverage Diff                  @@
##           feat/evals-framework    #3415   +/-   ##
=====================================================
  Coverage                 76.12%   76.13%           
=====================================================
  Files                      1201     1201           
  Lines                    115988   116035   +47     
  Branches                  10415    10421    +6     
=====================================================
+ Hits                      88294    88338   +44     
  Misses                    24671    24671           
- Partials                   3023     3026    +3     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.33% <6.38%> (-0.03%)	⬇️
OS-ubuntu-latest	72.17% <6.38%> (-0.03%)	⬇️
Py-3.10	72.17% <6.38%> (-0.03%)	⬇️
Py-3.11	72.17% <6.38%> (-0.02%)	⬇️
Py-3.12	72.16% <6.38%> (-0.03%)	⬇️
Py-3.13	72.17% <6.38%> (-0.02%)	⬇️
Py-3.14	72.17% <6.38%> (-0.03%)	⬇️
Py-3.14t	72.16% <6.38%> (-0.04%)	⬇️
SelfHosted-Large	29.68% <6.38%> (-0.01%)	⬇️
SelfHosted-Linux	35.94% <85.10%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/msgs/sensor_msgs/PointCloud2.py	62.01% <85.10%> (+2.78%)	⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.