Skip to content

[Dataset] Add BenchCAD (CAD code-understanding VQA) - #1583

Open
HaozheZhang6 wants to merge 1 commit into
open-compass:mainfrom
HaozheZhang6:add-benchcad-dataset
Open

[Dataset] Add BenchCAD (CAD code-understanding VQA)#1583
HaozheZhang6 wants to merge 1 commit into
open-compass:mainfrom
HaozheZhang6:add-benchcad-dataset

Conversation

@HaozheZhang6

Copy link
Copy Markdown

What

Adds BenchCAD, a benchmark for evaluating MLLMs on mechanical CAD parts, with two configs:

Dataset Input Output
BenchCAD_VQA rendered views of an industry-standard part a single number
BenchCAD_CodeQA rendered views + CadQuery source a single number

2,400 questions over 200 parts / 106 part families / 49 engineering standards (dimensions, counts, ratios).

Scoring

Deterministic, no judge model required:

  • exact match for counts / integers / yes-no
  • symmetric ratio accuracy min(pred, gt) / max(pred, gt) for dimensions and ratios

evaluate() reports an overall score plus a breakdown by question type. Mirrors the official BenchCAD scoring (https://github.com/BenchCAD/BenchCAD-main).

Data

Single TSV hosted on HuggingFace: BenchCAD/BenchCADvlmevalkit/BenchCAD_QA.tsv (md5 e19747288563fe807d0ceb8049da1989), images base64-encoded inline.

Testing

  • flake8 / isort / yapf (column_limit=120) all clean on the new file and the registry edit.
  • Unit-tested the scoring (_to_number, _score_one) and evaluate() aggregation against hand-computed values (overall/integer/dim/ratio), and build_prompt for both configs (VQA omits code, CodeQA includes the CadQuery block).

Notes

BenchCAD is described in arXiv:2605.10865 and is featured as an external benchmark in Anthropic's Claude Opus 4.8 system card (§8.16.4). Follows the ChartQAPro dataset pattern; one new file vlmeval/dataset/benchcad.py plus registry import/list entry in vlmeval/dataset/__init__.py.

Add BenchCAD, a benchmark for evaluating MLLMs on mechanical CAD parts,
exposing two configs:

- BenchCAD_VQA:    rendered views of an industry-standard part -> numeric answer
- BenchCAD_CodeQA: rendered views + CadQuery source -> numeric answer

2400 questions over 200 parts / 106 part families / 49 engineering standards.
Scoring is deterministic and needs no judge model: exact match for
counts/integers/yes-no, symmetric ratio accuracy min(p,gt)/max(p,gt) for
dimensions and ratios. Data is a single TSV hosted on HuggingFace.

Source: https://github.com/BenchCAD/BenchCAD-main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant