Rules.txt is missing from the open-source release

[din0s]

Dear MMDR-Bench team,

Thank you for releasing this benchmark! The multi-dimensional scoring approach seems very well thought out.

I ran into an issue when trying to use the evaluation pipeline on my own reports.
The GEN and EVI scoring prompts in detail/scoring_judge.py inject Rules.txt as context for the LLM judge:

# scoring_judge.py:137
prompt = f"""
You are a judge for General Metrics in a Deep Research benchmark.

Rules (excerpt):
{rules_text[:1400]}
...

But Rules.txt is not included in the repository. load_rules_text falls back to "No specific rules provided.", which means the judge receives no definitions for the fields it's asked to return (E, p_analysis, rho_connective, U, O, T, kappa, theta, support_strength, hallucination_prob, ...)

These fields feed directly into the scoring formulas (e.g., sigmoid(a0 + a1*TTR + a2*H - a3*L - a4*E) in scoring_general.py), so their interpretation matters. Without Rules.txt, a third party running the evaluation pipeline gets scores that depend entirely on whatever the judge LLM guesses these field names mean.

Could you include Rules.txt in the repo (or document the expected field definitions somewhere)?

[Ayanami-E]

I think I accidentally deleted it while trying to stay anonymous. Please wait for a moment. I'll upload it right away.

[18828098642]

Rules.txt.txt

Hi, this is the missing rules file

[din0s]

Thank you for your prompt response. From what I understand, these rules are primarily meant to guide the generation process, so that all baselines reported in the benchmark follow a similar structure and workflow.

A point that remains unclear to me concerns the GEN judge prompt (scoring_judge.py:137-176). The prompt asks the LLM judge to return a JSON object with several numeric fields, but provides no textual definitions of what each variable represents or what range/distribution each should follow, only an example JSON with sample values:

# scoring_judge.py:117-137 (schema) + :137-176 (prompt)
{
    "E": 0.05,
    "p_analysis": 0.40,
    "rho_connective": 0.60,
    "rho_synthesis": 0.50,
    "U": 0.70,
    "O": 0.65,
    "T": 0.75,
    "kappa": 0.02,
    "theta": 0.80,
    "llm_R": 0.70,
    "llm_I": 0.60,
    "llm_C": 0.65,
    "llm_S": 0.70
},

I can imagine that these variables have precise intended meanings (e.g., E = error rate, kappa = conflict rate, U/O/T = unity/ordering/transitions) and are consumed by specific formulas downstream. However, none of this context is communicated to the judge LLM in the prompt itself (unless I've missed it). The judge must infer the semantics purely from the variable names and example values.

Similarly, the evidence pair judge (scoring_judge.py:204) asks for support_strength, hallucination_prob, inversion_prob, and chain_sound_prob without defining them.

Shouldn't the judge prompts include explicit definitions of what each variable represents and guidance on the expected distributions? Without this, the scores seem highly sensitive to which LLM is used as a judge (as well as pure non-determinism), since different models/runs may interpret ambiguous variable names differently.

[18828098642]

Thanks for raising this. These variables are not meant to be free-floating natural-language concepts that each judge model has to invent from scratch. In our setup, the Rules file mainly specifies the required JSON interface, while the actual operational semantics and downstream usage are implemented in the Python evaluation pipeline. In other words, the judge is producing structured signals for code-defined metrics, rather than defining the metrics itself. More importantly, consistency is enforced at the pipeline level, not just at the prompt level. The benchmark converts reports into structured evidence objects, applies fixed downstream aggregation logic, and computes the final scores from those Python-implemented components. So the evaluation does not rely only on variable names or on a single LLM’s private interpretation of them. I agree that adding short natural-language glosses in the judge prompt could make the interface more readable and potentially reduce judge variance. But the current setup is not undefined: the semantics are already operationalized in code, and the consistency is established by the implemented scoring pipeline.

[din0s]

Thanks for the clarification, I think we're actually on the same page! I understand that the downstream aggregation logic is fixed in Python and that the pipeline doesn't rely solely on the judge's interpretation. That said, while looking at your related codebase here, I noticed that an earlier(?) version of the judge prompt did include explicit natural-language definitions for each variable:

Return a JSON object with the following floating-point fields in [0, 1]:

- "E": estimated proportion of obvious language errors or typos.
- "rho_analysis": proportion of paragraphs that are analytical (not purely descriptive).
- "rho_connective": normalized density of causal/contrastive discourse markers.
- "rho_synthesis": degree to which multiple pieces of evidence are compared and synthesized.
- "U": paragraph theme consistency.
- "O": explicitness of paragraph-to-paragraph transitions.
- "T": alignment of the conclusion with previous paragraph themes.
- "kappa": normalized rate of explicit contradictory statements (higher = more contradictions).
- "theta": structural ordering and section balance.

This is closer to what I was expecting. Were these definitions removed intentionally from the version in the current repository?

[18828098642]

Yes, that is basically what happened. Some earlier internal versions did include more explicit natural language definitions for these variables. During cleanup for the open source release, I removed quite a lot of inline explanatory material to keep the repository more compact and readable, especially in places where the operational definitions were already fixed by the Python pipeline.

So the omission was not meant to signal any change in semantics. It was mainly a presentation and cleanup decision. In retrospect, some of those short definitions were probably useful to keep, because they make the judge interface easier to read without changing the underlying implementation.

Apologies if that made the current version look more under specified than it actually is. This codebase went through several iterations, and with a project of this size it is not always easy to preserve every helpful explanatory fragment during refactoring and release cleanup. But your observation is fair. The earlier wording is consistent with the intended meanings, and keeping a concise version of those definitions in the current prompt or documentation would likely make the setup clearer.