LexCausa — Replication Package

Causality-Steered Dialectical Legal Argument Generation with Large Language Models.

This document is the replication guide for the accompanying paper. It explains how to (A) regenerate the experimental data and (B) reproduce the analysis (tables, figures, and the research-question results) from that data.

---

1. What LexCausa is

LexCausa turns the analysis of an Italian legal claim from plausible prose into an inspectable package. A claim is routed to a legal domain and grounded once in a shared evidential context retrieved from a Neo4j knowledge graph of statutes and precedents. Three agents then operate over that context:

• Reasoner — builds the thesis as a structured ASPIC+ chain (premise → statute → precedent → causal link → conclusion).
• Counter-Reasoner — builds the antithesis, selecting typed attacks from an explicit causal taxonomy (never inventing facts).
• Polisher-Evaluator — verifies every citation against the graph and adjudicates the confrontation with a deterministic Argument Quality Assessment (AQA) procedure, emitting a net plausibility verdict (plausible / implausible / uncertain).

The paper evaluates the framework with a full-factorial Design of Experiments over three factors — the model in the Reasoner seat, the model in the Counter-Reasoner seat, and the reasoning paradigm — plus a paired taxonomy ablation and a sensitivity analysis.

The campaign. 2 Reasoner models × 2 Counter-Reasoner models (cross-paired) × 3 paradigms × 22 claims × 5 replicas = 1,320 runs, produced on the OpenRouter backend with a fixed auxiliary model (llama_4_scout) for retrieval/classification/ evaluation. The released campaign lives in experiments/full_factorial/runs/.

Factor	Levels
Reasoner model	gpt_oss_120b, llama_3_3_70b
Counter model	gpt_oss_120b, llama_3_3_70b
Paradigm	Plan-then-Execute (on), Step-wise (off), Single-call (single)
Claims	22, over 4 domains (civil / criminal / administrative / mixed) — see claims.md
Replicas	5
Auxiliary + evaluator	llama_4_scout (fixed)

---

2. What is included in this package

experiments/
├── full_factorial/              # main full-factorial campaign (RQ1/RQ2/RQ3)
│   ├── runs/
│   │   ├── metrics.csv          # one row per run (1,320) — the analysis input (see §6)
│   │   ├── metrics.parquet      # same data, Parquet
│   │   └── runs/*.json          # raw per-run pipeline outputs (chains, attacks, citations)
│   └── analysis/
│       ├── main/               # doe_analysis.json + heatmaps (generated, see §B.1)
│       └── figures/            # paper figures (PDF, generated by lexcausa_stats.ipynb)
└── causal_taxonomy_ablation/    # taxonomy ablation (Appendix H) — A/B run outputs land here

scripts/
├── run_full.sh                   # one-command launcher: Neo4j → KB → backend → campaign → analysis
├── run_ablation.sh               # one-command launcher for the taxonomy ablation (Appendix H)
├── run_multi_doe.py              # campaign orchestrator (cloud backends + in-process vLLM / HPC via LLM_BACKEND, sharding)
├── run_doe_batch.py              # taxonomy-ablation A/B batch runner (→ experiments/causal_taxonomy_ablation/)
├── analyze_doe_results.py        # A/B ablation analysis (paired t-test, sign test, Cohen's d)
├── merge_doe_shards.py           # merge sharded/multi-machine runs into one dir
├── analyze_multi_doe.py          # statistical analysis → doe_analysis.json (RQ1/RQ2/RQ3)
├── sensitivity_tau.py            # verdict-threshold τ sweep over metrics.csv (App. G)
├── sensitivity_full.py           # ±25% perturbation of AQA scoring constants (App. G)
├── tune_aqa_with_gold_dataset.py # AQA calibration on gold-labeled verdicts (App. E.3)
├── tune_aqa_real_plus_synth.py   # AQA calibration on real + synthetic cases (App. E.3)
├── tune_retrieval_claims.py      # supervised retrieval tuning
└── token_report.py               # per-phase token accounting

lexcausa_stats.ipynb              # notebook that renders the paper tables/figures
claims.md                         # the 22 claims used in the campaign
claims_gold_labels.json           # gold labels for supervised retrieval / calibration
ablation_settings.json            # taxonomy-ablation A/B config (counter taxonomy on vs off)

---

3. Requirements & setup

Prerequisites: Python 3.11, Docker (for Neo4j), and — to regenerate data — an LLM provider key (OpenRouter for the paper backend). Reproducing the analysis only (Part B) needs neither Docker nor an LLM key.

3.1 Python environment (conda + pip)

conda create -n lexcausa python=3.11 -y
conda activate lexcausa
pip install -r requirements.txt

3.2 Environment variables

cp .env.example .env
# then edit .env and fill in your own values (Neo4j password, OPENROUTER_API_KEY, ...)

See .env.example for every variable and inline guidance. The keys that matter for the campaign are OPENROUTER_API_KEY, OPENROUTER_AUX_MODEL, LLM_MAX_TOKENS, and the Neo4j credentials.

---

Part A — Regenerate the experimental data

Requires Neo4j + a paid OpenRouter key. Skip to Part B to reproduce the analysis from the already-released metrics.csv.

A.1 One command (recommended)

scripts/run_full.sh runs the whole pipeline end to end: it starts Neo4j (a single Docker container), loads the knowledge base, launches the backend, runs the full 1,320-run campaign, and finally runs the analysis.

conda activate lexcausa                 # Python 3.11 env (see §3.1)
cp .env.example .env                     # then fill in OPENROUTER_API_KEY, NEO4J_* (see §3.2)

DRY_RUN=1 bash scripts/run_full.sh       # verify the run matrix (no LLM calls)
bash scripts/run_full.sh                 # run the full campaign + analysis

Useful overrides (all optional, set as environment variables):

Variable	Default	Meaning
WORKERS	1	parallel backends + DoE shards (merged automatically)
REPLICATES	5	replicas per condition
LLM_BACKEND	openrouter	openrouter / groq / local (vLLM)
FORCE_RELOAD	0	1 wipes and reloads the knowledge base
KEEP_BACKEND	0	1 leaves the backend(s) running afterwards
OUT / ANALYSIS_OUT	experiments/full_factorial/{runs,analysis/main}	output directories

Outputs: <OUT>/metrics.csv (+ raw JSON in <OUT>/runs/) and <ANALYSIS_OUT>/doe_analysis.json.

Part B — Reproduce the analysis

Everything below runs offline on the released metrics.csv — no Docker, no LLM key.

B.1 Statistical analysis (RQ1 / RQ2 / RQ3)

python scripts/analyze_multi_doe.py \
  --run-dir experiments/full_factorial/runs \
  --output  experiments/full_factorial/analysis/main

This writes doe_analysis.json (plus win/tie/loss heatmaps). The methodology matches the paper's Statistical Methods appendix:

• Factorial ANOVA — Reasoner, Counter, and paradigm main effects + the Reasoner × Counter interaction (RQ2), reported via the η² importance index (→ Table 2).
• Friedman + Dunn/Holm (RQ1) — omnibus test blocked by the 22 claims for each multi-level factor, with Holm-Bonferroni post-hoc as win/tie/loss matrices.
• Wilcoxon signed-rank — two-level contrasts blocked by claim (model-class contrast, RQ1; paradigm ablation, RQ3).
• Bootstrap CI — for citation fidelity (a proportion).

The three AQA dimensions (Cogency, NormSupport, Semantics) are analyzed separately in addition to the aggregate net plausibility.

B.2 Tables and figures

Tables. The paper tables come straight from the analysis of §B.1:

• Table 2 (factorial-ANOVA η² per factor) — the values are in experiments/full_factorial/analysis/main/doe_analysis.json under anova.<response>.effects.<factor>.eta_squared.
• Table 3 (per-level means by Reasoner/Counter model and paradigm) and the per-level descriptive statistics are rendered by the notebook lexcausa_stats.ipynb (below).

Figures. The paper figures (the Reasoner × paradigm interaction plot, the cost/quality frontier, the verdict distribution, etc.) are also rendered by the notebook lexcausa_stats.ipynb (in the repo root).

Everything it needs (matplotlib, seaborn, statsmodels, scikit-posthocs, and JupyterLab/nbconvert) is already in requirements.txt.

Its setup cell already points at the released campaign (RUN_DIR = experiments/full_factorial/runs, FIG_DIR = experiments/full_factorial/analysis/figures), so run it from the repo root — no editing needed.

Headless (recommended) — execute every cell and write the figures, no UI:

jupyter nbconvert --to notebook --execute --inplace lexcausa_stats.ipynb

Interactive — Jupyter Lab, then Run → Run All Cells (or open it in VS Code):

jupyter lab lexcausa_stats.ipynb

Either way the figures are written to experiments/full_factorial/analysis/figures/. The paper's Figure 3 (Reasoner × paradigm interaction on AQA net plausibility) is produced by the last cell as fig3_reasoner_paradigm_interaction.pdf. To point the notebook at a different run, edit RUN_DIR in its first cell.

B.3 Taxonomy ablation (Appendix H)

The taxonomy ablation is a separate paired A/B experiment — it is not part of the 1,320-run campaign (there the taxonomy is always on: causality_enabled = True in every row). Both roles are fixed to gpt_oss_120b; on each claim the thesis is generated once and frozen (aqa_lock_reasoner_plausibility: true), and the Counter-Reasoner runs twice:

• Setup A (baseline): counter_enable_causality = false — antithesis built without taxonomic guidance (no causal identity, no taxonomy-typed attacks).
• Setup B (treatment): counter_enable_causality = true — antithesis built with the full taxonomy.

Because the thesis is shared and frozen, the only systematic difference between the two arms is the taxonomic guidance to the opponent, which isolates the effect of the causal component (paper Appendix H). The configuration is ablation_settings.json.

Reproducing this ablation regenerates data — it needs Neo4j and an LLM key (like Part A), because it drives the pipeline over HTTP.

One command (recommended). scripts/run_ablation.sh does the whole ablation end to end: starts Neo4j, loads the KB, starts the backend, runs the A/B batch, and runs the analysis.

conda activate lexcausa
DRY_RUN=1 bash scripts/run_ablation.sh     # print the plan, run nothing
bash scripts/run_ablation.sh               # full ablation + analysis

Overrides (optional): REPLICATES (default 5), DOMAINS, ONLY=C1,P2 to restrict the claim set, RUN_NAME, KEEP_BACKEND=1. Output goes to experiments/causal_taxonomy_ablation/<RUN_NAME>/.

B.4 AQA calibration (Appendix E.3)

The AQA constants (weights, thresholds) were fixed by a staged grid search before the campaign, maximizing verdict macro-F1 on gold-labeled cases augmented with synthetic ones:

python scripts/tune_aqa_with_gold_dataset.py     # gold-labeled verdicts
python scripts/tune_aqa_real_plus_synth.py       # real + synthetic cases

Gold labels: claims_gold_labels.json. Retrieval tuning (Hit@k, MRR, nDCG@10) uses the same gold file:

python scripts/tune_retrieval_claims.py --top-k 30

B.5 Sensitivity analysis (Appendix G)

Two offline scripts confirm the paper's conclusions are robust to the AQA scoring choices. Both run on the released campaign, use only the standard library, and print their report to stdout (nothing is written to disk):

# τ sweep — vary the verdict threshold over metrics.csv and show the ranking is τ-stable
python scripts/sensitivity_tau.py  --run-dir experiments/full_factorial/runs

# full re-scoring — ±25% perturbation of the AQA weights (α,β,γ), damage factor (λ),
# and attack-type multipliers (m_t); reports Spearman ρ vs the default cell ranking
python scripts/sensitivity_full.py --run-dir experiments/full_factorial/runs

• sensitivity_tau.py sweeps the verdict threshold τ and reports the verdict distribution, the uncertain-band share, and the Spearman ρ of the design-cell ranking across τ — showing that τ only discretizes the (fixed) net plausibility and leaves the comparative ranking intact.
• sensitivity_full.py re-scores the stored per-run artifacts under ±25% perturbations of the scoring constants (the admitted-attack partition is held fixed), first validating the re-scorer against the stored aqa_plausibility, then reporting that the qualitative conclusions (instruction-tuned > native reasoner; Plan-then-Execute the top paradigm; Reasoner × Counter interaction η² ≈ 0) hold under every perturbation.

---

License

Released under CC BY-NC-ND 4.0. See LICENSE.md.