Suzanne

A judgment layer on top of Spot. Spot's geometric detector flags mesh vertices that are statistically out of place. But geometry alone cannot tell an accidental spike from an intentional one: the same spike is a defect on a smooth cheek and a feature on a horned demon. Suzanne renders each flagged region from several angles, sends the images plus the geometric evidence to a multimodal model, and receives a structured verdict - defect, feature, or uncertain. A fix is applied only to confident defect verdicts; everything else is surfaced to the human, untouched.

The product is not the judge. The product is the judge plus the benchmark that measures it. Without the benchmark this is demo-ware; with it, it is a system with a known error rate.

(Working name after Blender's mascot mesh - the monkey every Blender artist has inspected a thousand times - and a nod to the Suzanne Awards. Spot spots what is out of place; Suzanne rules on whether it was on purpose.)

The Blender add-on (H6): run Spot's detector on the mesh, judge each flagged region, fix only the confident defects - the whole pipeline over the untouched core.

---

Results

Measured on the synthetic benchmark (python -m suzanne.bench.run), prompt judge-v1, single run. Defect class: a smooth sphere with one injected spike per mesh, offset swept 0.5-4x the mean edge length (Spot's injector + detector). Feature class: clean meshes with genuine sharp geometry - cube, cone, beam, pyramid, capsule, and the real fandisk CAD model - where every flag is a geometric false positive whose correct verdict is feature. 57 cases (16 defect / 41 feature across 7 shapes).

Model (images)	Accuracy	Defect precision	Defect recall	Uncertain	Feature regions fixed at tau=0.8	$/case
gemini-2.5-flash	63.2%	55.6%	93.8%	14%	12 / 41	$0.0007
claude-opus-4-8	42.1%	55.2%	100%	35%	4 / 41	$0.0220
baseline: always-feature	71.9%	-	0%	-	0 / 41	-
baseline: always-defect	28.1%	-	100%	-	41 / 41	-

p50 latency ~ 7.5-7.9 s for both. What the numbers say, honestly:

1. Neither model beats the always-feature baseline (71.9%). Once the feature class is dominated by genuinely ambiguous hard-surface geometry (the capsule seam, fandisk's CAD chamfers), a borrowed semantic prior does not beat "assume every flag is a feature." That is the honest result on this set, reported as-is.
2. The metric that actually matters is the last column, and it inverts the ranking. Accuracy is a mislabel count; a suzanne fix only moves a vertex when the verdict is defect and confidence >= tau. Under the default tau=0.8, Gemini - the "winner" on accuracy - would wrongly move 12 of 41 feature regions, while Claude moves only 4. The abstainer destroys 3x less artist geometry despite scoring 21 points lower on accuracy. For a tool whose expensive error is smoothing an intentional feature, Claude is the safer choice - the opposite of what accuracy suggests. (Defect recall stays 94-100%, so this safety costs almost no missed defects.) The gate isn't airtight - the capsule reaches 0.83 > 0.8 - but it blocks most low-confidence mistakes. The asymmetric action policy, not the raw verdict, is the product.
3. "Which model is the conservative one?" has no stable answer - temperament is conditional on shape, not a trait of the model. In aggregate Claude abstains more (35% vs 14% uncertain), but per shape it flips: on the capsule Claude commits (11 of 12 -> defect) while Gemini hedges (5 defect / 7 uncertain); on fandisk it reverses - Claude hedges (12 of 15 -> uncertain) while Gemini commits (6 defect / 9 feature). Both share the same ceiling on ambiguous hard geometry; which one over-commits depends on the geometry, not the model.

Images earn their tokens. On an earlier image-vs-text ablation, stripping the images collapsed accuracy to the always-defect baseline - the geometric text alone cannot tell a feature from a defect. The rendered views added ~20 points. The difficulty curve over injected-spike magnitude is flat at 100% for Claude (93.8% for Gemini, which misses only the grossest spike): an isolated spike, subtle or gross, is a reliable defect.

The honesty arc. A first weak feature set (cube + cone, 9 cases) reported Claude at 84% / 100% precision - but it measured a problem the judge had already solved. Adding the capsule dropped it to ~55-64% and surfaced the real failure mode; adding real CAD (fandisk) dropped it again, below the trivial baseline. Each reinforcement measured a harder, truer problem. No cherry-picking - the numbers are whatever they are, and what they say is: this borrowed prior is a conservative assist gated behind a confidence threshold, not a solved classifier.

Adaptive escalation (H5)

For cases a model rules uncertain, --escalate renders two extra curvature-colormap views (hard edges glow, smooth regions stay cool), adds them to the evidence, and re-judges, then measures the accuracy gained against the tokens spent. (Measured on the earlier 42-case procedural set, before fandisk was added, so its baseline accuracies below are that run's 54.8% / 64.3%, not the 57-case headline above.)

Model	Uncertain	Resolved correct	Made worse	Still uncertain	Accuracy	Cost
claude-opus-4-8	8	6	1	1	54.8% -> 69.0% (+14.3 pts)	$0.21 / 2,950 tok
gemini-2.5-flash	8	0	3	5	64.3% -> 64.3% (+0.0)	$0.008 / 899 tok

The gain is shape-dependent, not model-dependent. Curvature views rescue abstentions on angular features - Claude's uncertain cube corners and beam edges resolved to the correct feature, because a clean regular sharp edge reads as deliberate. But on the capsule they backfire (four cases pushed from uncertain to a confident, wrong defect across the two models): the colormap makes the cap seam glow, and a glowing seam looks even more like a crease. The escalation amplifies "this is sharp", not "this is intentional" - so it helps where sharp means feature and hurts where sharpness is the ambiguity. It also has a real cost the benchmark surfaces: it converts honest abstentions into confident false positives (Claude 1, Gemini 3). And those are not just precision points: Gemini's three land at confidence

= 0.8, so they are three new wrong fixes the tau gate would let through and actually move (Claude's one is 0.70, below tau, so it stays harmless). The real cost of escalating is artist geometry moved, not just accuracy per token.

Honest caveats. The capsule contributes 12 of the 41 feature cases and its flags are symmetric duplicates (one seam sampled twelve times), so it over-weights a single failure mode; fandisk is capped to its 15 sharpest edges for the same reason. Read the per-shape story, not just the aggregate. Single sphere base for defects; single run; one prompt version.

---

How it works

Pipeline, per mesh (from Spot's JSON report):

1. Case builder - cluster adjacent flagged vertices into cases via connected components (flags within each other's 2-ring merge). One case = one judgment = one API call. This is the first cost lever.
2. Evidence renderer - per case, 4 images at 512px: three tight crops orbiting the region under strong key light (so displacement reads as shading), plus one context view of the whole object with the region tinted red.
3. The judge - one multimodal call per case. Two-step instruction (identify the object, then rule whether the local geometry is intentional for it), strict JSON output validated against a schema, retry once, downgrade to uncertain on a second failure. Tokens, dollars, and latency captured per call.
4. Asymmetric action policy - apply Spot's Taubin fix only where verdict == "defect" and confidence >= tau (default 0.8). Everything else is left byte-identical. Suzanne reuses Spot's correction (the report already carries each vertex's corrected_position), so a "fix" is just moving the gated vertices there.

Core purity rule. The core package (case builder, judge client, schemas, action policy) imports only numpy, scipy, pydantic, and the LLM SDK. trimesh and open3d live exclusively in the CLI adapter, so the Blender add-on (suzanne_blender/, H6) imports the core untouched - only numpy/scipy/pydantic + the SDK, all of which install into Blender's bundled Python - swapping in bmesh geometry and Blender's renderer. Enforced by a unit test.

---

Usage

Install (Python 3.11+, same environment as Spot):

pip install -e ".[llm,adapter,dev]"

Set ANTHROPIC_API_KEY (or run ant auth login); set GEMINI_API_KEY to use Gemini. The benchmark and the Blender review need the Spot repo on SPOT_PATH.

Judge & fix a real mesh - the product. No injection: Spot's detector flags the mesh, Suzanne rules on each flag, and only confident defects are fixed.

# 1. Spot flags the statistically-out-of-place vertices (upstream dependency)
python -m spot fix model.obj --out fixed_by_spot.obj --report report.json

# 2. Suzanne judges each flagged region
#    --limit N   -> judge only the N most suspicious (cheap inspection of a big mesh)
#    --model-id gemini-2.5-flash -> use Gemini instead of Claude
python -m suzanne judge model.obj --report report.json --out verdicts.json

# 3. Suzanne fixes ONLY confident defects; every other vertex stays byte-identical
python -m suzanne fix model.obj --report report.json --tau 0.8 --out fixed.obj

Benchmark - builds both ground-truth classes with Spot, judges, and reports the confusion matrix, difficulty curve, baselines, and cost:

python -m suzanne.bench.run out/ --ablation                                # Claude + no-image ablation
python -m suzanne.bench.run out/ --models claude-opus-4-8,gemini-2.5-flash  # model side-by-side
python -m suzanne.bench.run out/ --feature-dir meshes/feature              # + real CAD meshes (fandisk)
python -m suzanne.bench.run out/ --escalate                                # H5 curvature escalation

Blender add-on - review & fix inside Blender, over the untouched core. Build the installable zip with python scripts/package_addon.py; full install + usage in suzanne_blender/README.md.

---

Positioning

Suzanne is deliberately narrow, and it sits next to a lot of existing work:

• Asset-level 3D evaluators score whole generated assets for quality. Suzanne is local: one flagged region, one ruling - not a whole-asset score.
• Mesh-as-text LLMs (LLaMA-Mesh, MeshLLM) serialize geometry into an LLM's context; that caps out at toy face counts. Suzanne never sends vertices as text - evidence goes in as rendered images of local crops.
• GNN mesh denoisers (GeoBi-GNN, DOGNET, ADMD) and classic feature-preserving filters remove noise while trying to preserve edges, from geometry alone. Suzanne does not denoise; it adjudicates intent using a borrowed semantic prior, then gates a surgical reuse of an existing geometric fix.

What Suzanne adds: local adjudication of detector flags with a borrowed multimodal prior, measured on a synthetic benchmark, gating a conservative fix - with the measured numbers above, whatever they are.

---

Status

Milestone
H0	Evidence harness (case builder + renderer)	✅
H1	Single-case judge (structured verdict, retry, telemetry)	✅
H2	Benchmark (both classes, confusion matrix, difficulty curve, cost)	✅
H3	Gated fix + CLI (judge / fix, byte-identical output)	✅
H4	Multi-model (Gemini side-by-side table)	✅
H5	(stretch) adaptive escalation on uncertain verdicts	✅
H6	(stretch, post-V1) Blender adapter over the untouched core	✅ (validated in Blender: the Suzanne monkey, spiked, judged & fixed)

Not in scope: model training or fine-tuning (the semantic prior is borrowed, never learned here); a claim of solving intent understanding (only measured accuracy on a synthetic benchmark is claimed); autonomous action under uncertainty (uncertain/feature always leaves the mesh untouched).

License

Apache 2.0.