feat(evals): generate simulator-grounded VQA cases from RoboCasa

[Moshiii]

Contribution path

• Linked foundation PR: feat(evals): passive/interactive agent eval framework over memory2 #3411
• Branch note: this PR is intentionally opened as a stack on feat/evals-framework (feat(evals): passive/interactive agent eval framework over memory2 #3411), because main does not yet contain the evaluation framework used by this feature. The stacked development branch is retained for now, with the VQA work isolated in one clean feature commit (c23561c58). After feat(evals): passive/interactive agent eval framework over memory2 #3411 merges, I will make this a truly independent branch by running git rebase --onto origin/main 0ddf8b493 feat/vqa-generation-framework and force-pushing the rebased branch.

Problem

#3411 adds the evaluation framework and its Memory2-backed passive runner, but image VQA cases still need to be authored manually. A developer has no reproducible path from a simulator use case to rendered observations, grounded questions, reference answers, and a case manifest that the existing dimOS evaluation runner can consume.

Solution

Add a small VQA generation layer that runs RoboCasa through an external Python environment and converts seeded simulator scenes into the existing dimOS evaluation inputs.

The generator exports RGB and instance-segmentation observations, normalizes simulator metadata into source-independent scene facts, and produces two initial question families: semantic presence and spatial left/right. Answerability gates reject ambiguous or visually weak cases using visibility, pixel area, label uniqueness, horizontal separation, and bounding-box overlap. Family balancing and deterministic seeds keep generated datasets reproducible.

The output contains a versioned cases.json manifest and a Memory2 observations.db, so it can be loaded directly by the existing passive evaluation path. RoboCasa, robosuite, and MuJoCo remain outside the dimOS runtime; this change adds no production dependency and avoids coupling the core package to a simulator-specific environment.

Developer workflow:

dimos evals generate use_case.yaml --source-python /path/to/robocasa/python
dimos evals run generated/use_case/cases.json

This PR deliberately does not add a new evaluation engine, scorer, agent loop, or generalized scene generator. It provides the necessary adapter and schema boundary for simulator-grounded VQA generation, with two concrete families that can be tested end to end.

How to Test

uv run pytest dimos/evals/test_generate.py dimos/evals/test_robocasa.py -m 'not (self_hosted or mujoco)'

ROBOCASA_PYTHON=/path/to/robocasa/python \
  uv run pytest dimos/evals/test_robocasa.py -m 'self_hosted and mujoco'

uv run mypy \
  dimos/evals/cli.py dimos/evals/generate.py \
  dimos/evals/robocasa.py dimos/evals/vqa.py

uv run ruff check \
  dimos/evals/cli.py dimos/evals/generate.py dimos/evals/robocasa.py \
  dimos/evals/test_generate.py dimos/evals/test_robocasa.py dimos/evals/vqa.py

uv run ruff format --check \
  dimos/evals/cli.py dimos/evals/generate.py dimos/evals/robocasa.py \
  dimos/evals/test_generate.py dimos/evals/test_robocasa.py dimos/evals/vqa.py

Local verification completed with 32 relevant evaluation tests passing, including a real RoboCasa/MuJoCo fixed-seed run. Seeds 100-102 generated 18 accepted cases: 14 semantic-presence cases and 4 spatial left/right cases. Eight repository static checks also passed.

Checklist

• I have read and approved the Contributor License Agreement.