Some details about the evaluation

[Helen1p]

Thanks to the authors for this great work. I have a few concerns:

• The reference image for Image similarity metrics (e.g., SigLIP-2/DINOv3) are obtained from the original Infinigen canonical renders with full materials/textures, while the system prompt explicitly instructs models to “Leave geometry untextured.” Therefore, under ideal instruction following, the generated output should be geometry-only, creating an inherent mismatch.
• At the same time, prompt_description itself contains appearance/texture-related descriptions, which conflicts with the system_prompt to leave geometry untextured. Due to this ambiguity and imperfect instruction following, many models may nevertheless generate materials/textures. This failure can accidentally make the prediction better aligned with the full-material reference images, thereby increasing image-similarity scores.

[gaoypeng]

Hi Helen,

Thanks for pointing out the issues. Yeah, I meant to add the prompt "leave geometry untextured" to the system prompt for a fair comparison for human voting (A/B testing), since I found that humans prefer a 3D object with better texture or materials, even if it has some floating parts. I also noticed prompt conflicts. Therefore, for all the human votes, the frontend UI in the 3D Code Arena will delete the texture/materials. The same policy also applies to computing image similarity metrics, which are calculated between (textured) reference images and (untextured) output objects generated by LLMs/VLMs/coding agents to ensure a fair comparison.

The lack of texture evaluation is one of 3DCodeBench's limitations and hasn't been fully studied yet. The ability to generate texture/material via code is also important. Calculating metrics between colorful reference images and untextured renders also shows a mismatch. I would continue to improve the benchmark further, and I'd welcome your contributions/ideas on how to evaluate the texture/materials.