diff --git a/docs/research/pdf-backend-bakeoff/RESULTS-EXTERNAL-VALIDITY.md b/docs/research/pdf-backend-bakeoff/RESULTS-EXTERNAL-VALIDITY.md new file mode 100644 index 00000000..b7034b02 --- /dev/null +++ b/docs/research/pdf-backend-bakeoff/RESULTS-EXTERNAL-VALIDITY.md @@ -0,0 +1,199 @@ +# Results: external-validity run — VOID + +- Status: **the run is void under its own pre-registered controls.** It produces no + comparative architecture result and none should be read out of it. +- Population: 17 confirmatory-holdout documents, 4190 pages, frozen before execution. +- Protocol: [`validation/external-validity/PRE-REGISTRATION.md`](validation/external-validity/PRE-REGISTRATION.md). + Deviation register: [`validation/external-validity/results/DEVIATIONS.md`](validation/external-validity/results/DEVIATIONS.md) + (entries through A62.4). + Scored output: [`validation/external-validity/results/metrics.json`](validation/external-validity/results/metrics.json). + **Every figure below is derivable from those two files.** +- Supersedes nothing in [`RESULTS.md`](RESULTS.md) or + [`RESULTS-CONFIRMATORY.md`](RESULTS-CONFIRMATORY.md). This run was designed to test + whether their findings generalise to unseen documents. **It could not, because the + instrument failed, not because the finding failed.** +- Input to [DeltaTrack#112](https://github.com/AgoraDMV/DeltaTrack/issues/112). + +--- + +## Headline + +**§5.6's N-B control failed, and §5.6's consequence for that is "run void".** + +The study did not measure the architectures. It measured its own adjudication instrument, +and the instrument did not survive. The useful finding is about the instrument. + +--- + +## What voided it + +Controls require both adjudication routes to pass on every fixture, with no tolerance. + +| control | what it tests | AI route | human route | combined | status | +|---|---|---|---|---|---| +| **N-A** | injected defects are transcribed, not repaired | 7/8 | **3/8** | 10/16 | **FAIL** | +| **N-B** | unambiguous XML-corroborated headings agree | **8/8** | **6/8** | 14/16 | **FAIL** | +| **N-C** | no heading reported where none is printed | 4/4 | 4/4 | 8/8 | PASS | + +§5.6: N-A failure voids M2/M3; **N-B failure voids the run**, because it shows the +adjudicator is unreliable independently of any architecture. + +**Every N-B failure and five of six N-A failures are on the human route.** The AI's single +N-A failure is fixture `ab2f34a86416bf77`, where *both* routes failed identically — a +strike-through bridges the injected gap so the defect is not legible in the render. That is +a fixture defect, tracked as issue #727. Setting it aside, the AI route is **N-A 8/8, +N-B 8/8, N-C 4/4**. + +### The human route repairs the defect class the study exists to measure + +Both failure modes are systematic, not noise: + +| human-route behaviour | evidence | +|---|---| +| **normalises spacing defects** | DELETE_ONE_WORD 3/3, but **WELD 0/3** and **SPLIT 0/2**. `OPERATIONSAND SUPPORT` transcribed `OPERATIONS AND SUPPORT`; `OPERA TIONS AND SUPPORT` transcribed `OPERATIONS AND SUPPORT`. | +| **substitutes a familiar spelling** | `RESCISSION` read as `RECISSION`. This is both N-B failures. The AI route is 4/4 correct on the same word. | + +Three of the five spacing fixtures carry no strike-through and render cleanly at 300 DPI. +The defects were legible; they were read through. **Weld and split are precisely the +defect classes the bake-off exists to detect, so the ground truth silently repaired the +thing being measured.** That is why the failure is disqualifying rather than a tolerance +question. + +On the 25-stimulus C-audit where both routes answered the same images, they agree exactly +on text and role in **16 of 25**. Of the 9 differences, 4 are the designator question +below, 2 are role-only, and 3 are human transcription or segmentation errors confirmed +against the rendered images. On all three the AI route matches the image. + +--- + +## The second instrument finding: the heading definition is underdetermined + +The frozen adjudicator prompt does not settle whether running page furniture is a heading, +and the gap is load-bearing. + +Its definition is compositional — "centered, or set in capitals, or set in italic, or set +in a distinctly larger or heavier face". Its first listed property is *centered*, and a +centered page folio is centered, is typographically separated, and is not one of "the small +numbers printed in the left margin". **Read strictly, the test makes page numbers +headings.** No one read it that way: the AI notes discuss a page folio in 25 answers and +report one as a heading in none, and across all 167 committed answers neither route ever +reports a bare-number heading. + +So every reader applies an unstated page-furniture carve-out that the text never defines. +It excludes folios on everyone's reading. Whether it also excludes a page-foot bill +designator (`•S 1609 PCS`) is exactly what the instrument does not say — and the routes, +and the AI route internally, went both ways: + +| page furniture class | admitted as a heading | excluded | consistent? | +|---|---|---|---| +| centered page folio | 0 | 25 | **yes** | +| page-foot bill designator | 23 | 2 | **no** | + +This surfaced through the R1 test-retest control, which failed at 4/5 against a 0.90 +threshold. The failing pair is one region presented twice at visually equivalent scale. +**Both presentations print the designator and the adjudicator saw it both times** — its own +note on the one it declined reads "set in bold and separated from the text block, but it is +a page footer/document identifier rather than a heading over any content". One adjudicator, +one frozen prompt, two incompatible rules. Full analysis: register entry A62.4. + +Roughly three in ten of the headings in the C-frame's adjudicated enumeration (19 of 65) +are running page furniture whose status the instrument does not settle. + +--- + +## Why RQ1 and RQ2 have no answer + +**RQ1 (comparative) — no evidence base.** RQ1's primary frame is the D-frame, a census of +every region where the two architectures' output differs, and the design required *every +D-frame item human-adjudicated*. The D census is 13,992 regions. Forty-five human answers +exist in total, of which 25 are the C-audit. The human arm was never executed at census +scale; the A27.3 budget and its A48 implementation resolved D-only items to no required +route. **RQ1 was foreclosed before the controls ever ran**, and the control failure is a +second, independent reason it cannot be answered. + +**RQ2 (absolute) — computed, then voided.** The C-frame produced figures, but M2 and M3 are +void under §5.6 via the R1 text failure, and the whole run is void via N-B. Recorded for the +register only, not to be cited: + +| metric | value | status | +|---|---|---| +| M1 precision | 38/42 = 0.905 | reported, run void | +| M1 recall | 38/65 = 0.585 | reported, run void | +| M2 exact heading text | 10/38 = 0.263 | **void** (R1 text FAIL) | +| M3 clean rate | 10/65 = 0.154 | **void** (R1 text FAIL) | +| M5 role agreement | 35/36 | reported, run void | + +**The C-frame cannot separate the architectures in any case**: H and X produce identical +values on every pooled C metric. Cross-engine qualification returned +`both_headlines_qualified: false` with no document-level qualification for either RQ. + +Every figure in this run is additionally labelled NON-CONFIRMATORY under §4.7, because +post-boundary deviations (A45, A48) were made after results were visible. + +**The population was not the problem.** §4.5 adequacy returned **GENERALISABLE**: 2583 +heading occurrences of the counted kinds across 7 filled strata, well above the +5-strata / 300-occurrence floor. S1 fired, all 17 documents scored, and the extraction ran +clean across all 4190 pages. **What failed was adjudication**, which is why a bigger or +different document sample would not help. + +--- + +## What is worth keeping + +1. **An AI image-adjudicator outperformed human ground truth on this study's own controls** + (8/8 vs 6/8 on N-B; 8/8 vs 3/8 on N-A excluding the illegible fixture). The study was + designed on the assumption that human adjudication is the reference. On the defect classes + the controls probe, that assumption did not hold. +2. **Human adjudicators read for meaning.** They normalise spacing and correct spelling + without noticing. Any future protocol that uses human transcription as ground truth for + character-level defects needs a control that can catch this, and needs to expect it. +3. **A compositional heading definition needs an explicit page-furniture rule.** A successor + instrument should settle running headers, footers, folios and bill designators in one + sentence rather than leave them to be inferred. +4. **M3's boundary census found 445 TEXT_ERROR but zero WELD and zero SPLIT** across 975 + outcomes. Given that the human route demonstrably normalises welds and splits away, this + is worth a successor's attention. It is flagged as a question, not a conclusion. + +## What is not worth doing + +Re-running the human arm. Both failure modes are systematic rather than noisy, so a larger +human sample fails the same way. Rescuing RQ2 means re-adjudicating a fresh population under +a new authorization, at significant cost, with no reason to expect a different result. + +## Known open defects, not exercised by this void run + +- **#726** — the scorer emits `NOT_EVALUABLE_NO_R1_PAIRS` while `rule3_gates` accepts only + `NOT_EVALUABLE`, so a zero-evidence R1 falls through to PASS. Fail-open. Not triggered + here, since R1 failed on a non-empty denominator. **Fix before any successor run.** +- **#727** — the N-A fixture above whose injected defect is not legible in the render. Its + `mutation_evidence` validates the text layer only and asserts nothing about legibility. +- **G2** — the baseline fixture is already invalid, so several G2 controls pass for reasons + unrelated to their labels. +- **No canonical decision operation.** `score_metrics.score_canonical()` loads committed + artifacts and writes `metrics.json`; there is no equivalent on the decision side. + `decide_architecture.decide()` exists, but every `DecisionInputs(...)` construction site is + a control probe with supplied facts, and `results/scores.json` is never written. Not built + here deliberately: the void verdict is already machine-computed and committed in + `metrics.json` `control_verdicts`, so a decision module would re-emit a foregone conclusion. + +## Reproduction + +**On this branch.** `results/metrics.json` and the register through A62.4 are included, and +every figure in this document is derivable from them: control verdicts and the per-fixture +failures from `control_verdicts`, the R1 pair facts from `r1_reliability`, M1–M5 from +`headings_pooled.C`, the population verdict from `adequacy_4_5`. + +**Not included, deliberately.** `results/oracle_adjudicated.json` and `ORACLE-PROVENANCE.json` +carry the adjudicated answers and are mode 0600 in the working tree. Publishing them would +expose answer-key material to any successor run, which the study's own contamination +apparatus exists to prevent. Re-deriving the raw per-answer counts quoted in A62.2 and A62.4 +(designator and folio tallies, the 16/25 C-audit agreement) therefore requires the study +branch. + +**The full apparatus** — the 307 MB answer key, `frames.json`, and the 15,437 rendered +stimuli — lives on the local branch `pdf-study-continuation-execution`, which is **not pushed** +because it tracks `oracle_key.json`. Preserve it as a bundle in `delta_track_mirror`. + +Gate state at close: `x04_freeze_check` reports freeze integrity COMPLETE, 19/19 checks pass, +and `EXECUTION FORBIDDEN` pending a sequence-9 authorization that was deliberately not minted, +because nothing further is to be scored. diff --git a/docs/research/pdf-backend-bakeoff/validation/external-validity/results/DEVIATIONS.md b/docs/research/pdf-backend-bakeoff/validation/external-validity/results/DEVIATIONS.md index 651bcf68..6ab40389 100644 --- a/docs/research/pdf-backend-bakeoff/validation/external-validity/results/DEVIATIONS.md +++ b/docs/research/pdf-backend-bakeoff/validation/external-validity/results/DEVIATIONS.md @@ -1190,3 +1190,482 @@ detection breaking. **What A56 does not do.** It changes no gate, no authorization, no scientific value, and no research artifact. It does not integrate any branch, create an authorization, prepare a packet, adjudicate, score, or decide the architecture. + +## A57 — POST-BOUNDARY APPARATUS DEVIATION + +```json +{ + "id": "A57", + "kind": "DEVIATION", + "commits": [ + "dcc2e990", + "49ed1f7d", + "7e87acd2", + "b263360f", + "fc8ce91f", + "8a3ad000", + "3321d5b6", + "fafa2b45", + "4b7b851e", + "05e772e2", + "a5fa991a", + "39359aae", + "17dfcd72", + "f0687016", + "0a5d9eb3", + "ce717732", + "ba7cae8d", + "38c0982e", + "a398eaa6", + "202b7ec9", + "57e73d72" + ], + "classification": "POST-BOUNDARY APPARATUS DEVIATION (AI ADJUDICATION ARM)", + "made_after_boundary": "de60dddf906bc4b01e5ffbe9af4d3e833a9a2be7 (continuation boundary)", + "results_already_visible": { + "members": 17, + "pages": 4190, + "d_frame_census": 13992, + "s1_documents_firing": "17/17", + "p_head_documents": 12, + "p_head_pages": 2864, + "cross_engine": "17/17 measured, n_qualified 0" + }, + "affects_membership": false, + "affects_scoring_rule": false, + "affects_metric_values": false, + "affects_architecture_decision": false, + "affects_execution_authorization": true, + "affects_reproducibility_surface": true, + "narrowing": "A57 BUILDS THE AI ADJUDICATION ARM that PRE-REGISTRATION 5.5 specifies and no code implemented. It reads no holdout byte and changes no threshold, selection rule, metric definition, route derivation or architecture rule. It does not modify the population, the frames, the frozen key, the blind artifact, the images, the original execution marker or either existing authorization. affects_metric_values is FALSE: no metric definition moves, and the single change to a pre-existing surface file (score_metrics.write_metrics at 3321d5b6) only REFUSES -- it computes nothing new and alters no value that was computable before. affects_execution_authorization and affects_reproducibility_surface are TRUE by construction: three files join METHODOLOGY_SURFACE / RESULT_BEARING_DATA, taking the authorization manifest from 31 entries to 34, so sequence 2 correctly goes stale and cannot speak for files it has never seen. The scorer's frozen import allowlist is widened by one, `ai_finalize`, so that whether an oracle may be scored can depend on where it came from; x27's own control is what made that widening visible, and the rejected alternative (moving verify_provenance into build_oracle) is recorded at the allowlist. NAMING NOTE: commit messages under these shas refer to 'the A57, A58, A59 and A60 reviews'. Those were labels for four independent reviews, not register ids; this record is the first use of A57 as a deviation id and the reviews have no register entries. COMMIT COVERAGE EXTENDED AFTER THE SURFACE GREW. results/AI-ADJUDICATION-PROTOCOL.json was not result-bearing when it was written, so the six commits that touched it (4b7b851e, 05e772e2, 49ed1f7d, a5fa991a, 39359aae, 17dfcd72) named no surface path and needed no declaration. Adding it to RESULT_BEARING_DATA at fafa2b45 made its entire history result-bearing, and surface_provenance_errors correctly required every one of those commits to be declared and to name the path. They are declared here rather than exempted: a file whose history is unaccounted for is exactly the silent surface hole A50 exists to close. 17dfcd72 SIMPLIFIED the arm, removing the run ledger, commencement semantics, resume-only recovery, run states and the two-phase START marker -- roughly 1,500 lines defending a property no local mechanism can enforce, now attested instead. It also carries three repairs to the GATE itself: blob_sha treating a path absent at a commit as present, the A50 synthetic controls inheriting the real RESULT_BEARING_DATA, and the self-test's membership tamper not surviving a signal. f0687016 rebuilt ai_finalize against the bundle the simplified runner actually writes -- it had been left addressing the removed run- layout, so it could not have run at all -- and made derivation REPLAY the runner's acceptance rules over raw stdout rather than read the `accepted` flag beside it. 0a5d9eb3 replaced score_metrics.write_metrics(payload, path) with score_canonical(), which loads the committed inputs, verifies the oracle's origin, scores and writes as ONE operation: the payload argument made 'verify this oracle, write a metric computed from that one' expressible, which the protocol's finalizer.scoring_gate forbids. ce717732 extended the replay to compare every field an attempt records about itself against its own raw stdout, not only `accepted`, because the transcript is what a reviewer reads and an unchecked summary field can misdescribe an attempt without moving an answer. ba7cae8d drove `materialize()`, the production entry, which was the one path in the arm with no control over it -- the same shape as the f0687016 defect, where a module was green over a layout nothing produced. It changes no result-bearing code: x34_ai_finalize.py is a control file, declared here because F9 protects every .py under the study directory, not because it is on the authorization surface. NO METRIC DEFINITION MOVES IN ANY OF THEM: score() is untouched, and affects_metric_values stays FALSE. The scorer's frozen import allowlist is widened by a further two, `execute_study` (the committed owner of which documents are in the study, so that canonical_inputs does not become a second authority for the population) and `hashlib` (the oracle digest that binds the bytes verified to the bytes scored); the rejected alternative for each is recorded at the allowlist, as the ai_finalize widening was. ANSWERED AN INDEPENDENT ADVERSARIAL REVIEW AT 38c0982e, which found seven real defects in the arm as it stood at sequence 3. Four were introduced this session. In order of consequence: a harness envelope decoding to anything but an object raised after the paid subprocess call and before the transcript write, destroying the evidence of an attempt already paid for (predates this session, and would also have broken derivation, which replays the same function over committed stdout); the canonical score read oracle_key.json, s1_control.json and cross_engine_control.json straight off disk, so an edited S1 or cross-engine row moved a reported result without touching the AI bundle; provenance named the run directory by absolute path, defeating the clean-checkout reproduction the protocol's evidence bundle requires; the runner that produced a bundle was never compared with the runner replaying its acceptance rules, so a rule change was silent because it need not change any answer; the digest binding verification to scoring was taken by re-opening the file after verification returned; materialize() inverted the protocol's attestation order and checked nothing, so the statement would be signed by someone who had already seen the result; and several provenance fields were written and never read, letting the record assert more than a green verifier established. NO METRIC DEFINITION MOVES IN ANY OF THEM EITHER: score() is still untouched and affects_metric_values stays FALSE. probes/execute_study.py joins files_touched because the committed-input check belongs with load_frames, which has asked the same question since A43; x27's own import-allowlist control is what established that, by refusing the first version that asked it inside the scorer. A SECOND ADVERSARIAL PASS OVER THOSE FIXES, answered at a398eaa6, found more -- and the worst of it in the RUN path rather than the analysis. A usage limit discarded attempts that had already been paid for and handed the item a fresh retry budget on resume, so a stimulus could receive more attempts than the protocol declares; transcript writes were not atomic, so an interruption could leave a file whose name said the item was finished and whose contents could not be parsed; the runner identity was hashed when RUN.json was written at the end rather than before the first paid call, so a runner edited mid-run would be recorded as the one that produced the bundle; the oracle was written WITHOUT build_oracle's assert_write_permitted, making it the one canonical artifact producible before a valid execution boundary; the attested bundle digest was not compared with the materialized one; and the committed-input check was check-then-read rather than one read compared against HEAD, which also left a tracked-symlink hole. Four controls were found to be false-green and were repaired so each kills the code it names. NO METRIC DEFINITION MOVES HERE EITHER: score() remains untouched and affects_metric_values stays FALSE. THE ONE FINDING NOT FIXED is recorded rather than smoothed away: committing results/AI-ADJUDICATION-ATTESTATION.md trips F9, which requires a declaration, which takes the standing authorization stale. That is the gate working as designed, and adding the attestation to F9_IGNORE would weaken a gate for the operator's convenience; the sequence belongs in the runbook instead. A THIRD REVIEW, RUN IN TWO HALVES, RETURNED NO-GO ON BOTH, answered at 202b7ec9. The most serious finding was that the oracle could be written with NONE of its preconditions: they lived in materialize(), while finalize() writes the canonical artifact by default, so calling the lower function with an injected runs_root and item list produced the study's oracle from an external uncommitted bundle -- and verification passed it, because it checked the item COUNT and never that the ids were the frozen AI-route population. Next worst was in the arm's own acceptance rule: a refusal the model expressed AS JSON beside an empty heading list was accepted as the substantive claim 'no heading is printed in this image', while the same refusal in prose was correctly refused -- a defect that converts a refusal into a metric denominator. The run loop lost paid attempts on interruption and granted a fresh retry budget, did not stop on a usage limit reported through the raw error channel, and re-read the prompt per item after checking it once. THE PROTOCOL IS AMENDED under this entry, with the operator's approval and BEFORE any stimulus was adjudicated: it now declares the three model-identity failure conditions the code enforced and the protocol omitted, the six isolation control labels a committed bundle is held to, and an atomicity claim corrected to what the code does. That amendment is legitimate exactly as the protocol's own amendment_note says -- the ai namespace is empty and no transcript exists, so the artifact still records what was fixed before anyone could see a result. NO METRIC DEFINITION MOVES: score() remains untouched and affects_metric_values stays FALSE. THE ARM WAS THEN EXECUTED, and 57e73d72 is the commit the note above anticipated: it carries the operator's signed attestation naming run id bf3e1346d963647071b996af73d94785e8727a97e05cbaf29a850f5ca13f75d7, it is a post-freeze .md under the study directory, and it is declared HERE rather than added to F9_IGNORE, with the authorization regenerated behind it. The run itself is committed at 0a6cdd02 as results data, which F9 does not protect and which therefore needs no declaration: 122/122 items answered, every one on its first attempt, no refusals, no usage-limit stop, all six declared isolation controls passing with their leaking arms, and model_usage naming only the declared confirmatory model and the declared auxiliary. NO METRIC DEFINITION MOVES HERE EITHER: score() remains untouched and affects_metric_values stays FALSE.", + "files_touched": [ + "probes/ai_adjudicate.py", + "probes/ai_finalize.py", + "probes/execute_study.py", + "probes/score_metrics.py", + "probes/x04_freeze_check.py", + "probes/x27_score_metrics.py", + "probes/x33_ai_adjudicate.py", + "probes/x34_ai_finalize.py", + "probes/x35_mutation_harness.py", + "results/AI-ADJUDICATION-ATTESTATION.md", + "results/AI-ADJUDICATION-PROTOCOL.json" + ] +} +``` + +**What was missing.** PRE-REGISTRATION 5.5 specifies the coverage-frame oracle as AI +image-adjudication by a separate `claude -p` process with no repository context, its prompt +and transcript committed. No code implemented it. The human route was adjudicated and +committed at 0b493c9f; the AI route's 122 items had never been run, and +`score_metrics.validate_inputs` refuses without them, so the study could not be scored. + +**Why it took four reviews.** Three independent code reviews each found the same property — +that the first binding run cannot be re-rolled — unenforced in a different place: no driver at +all, then a driver gating only the cheap setup phase, then a guard bypassable by deleting a +directory, aborting mid-run, or running twice concurrently. Each time the author reported it +implemented and verified beforehand. A fourth review of the APPROACH found the rule itself was +wrong rather than merely unimplemented: "the first run that COMPLETES binds" permits watching +transcripts land item by item and aborting once they look disagreeable. The rule now binds on +first COMMENCEMENT with resume-only recovery. + +**What is claimed, and what is not.** The binding property is an ATTESTATION, not a repository +proof. The operator controls the filesystem, the processes, the credential and git history, so +no local mechanism can establish that a command was never run; local guards defend against +ACCIDENT and the operator's signed attestation carries the rest. Assurance beyond that needs an +executor the operator does not control, which is a study-design decision and not a coding one. +The execution marker already makes the same admission in the same words. + +**Isolation is measured, not assumed,** and the finding was counter-intuitive: `--restricted` +suppresses BOTH project-level and user-level memory, while `ANTHROPIC_CONFIG_DIR` alone does +not — with it set and `--restricted` absent, a planted project `CLAUDE.md` was read back +verbatim and a separate probe recovered the operator's name and role. Every isolation control +has an arm that must leak, and each was proved to fire before its protected counterpart was +trusted. + +**Verified.** x33 51/51, x34 24/24, x27 195/195, x35 23/23. x35 is a mutation harness: it +breaks the apparatus on purpose, 23 defects one at a time, and requires the verifier to notice. +Its first run caught 10 of 22 — the number that justified the exercise, since x33 had until +then reported 26/26 while staying green under a deleted binding guard. It is a corpus, not a +proof: it establishes that these specific defects cannot pass unnoticed. + +## A61 — POST-BOUNDARY APPARATUS DEVIATION + +```json +{ + "id": "A61", + "kind": "DEVIATION", + "commits": [ + "f865dfb4" + ], + "classification": "POST-BOUNDARY APPARATUS DEVIATION (AUTHORIZATION GENERATOR REPAIR)", + "made_after_boundary": "de60dddf906bc4b01e5ffbe9af4d3e833a9a2be7 (continuation boundary)", + "results_already_visible": { + "members": 17, + "pages": 4190, + "d_frame_census": 13992, + "s1_documents_firing": "17/17", + "p_head_documents": 12, + "p_head_pages": 2864, + "cross_engine": "17/17 measured, n_qualified 0", + "ai_route": "122/122 adjudicated, committed at 0a6cdd02" + }, + "affects_membership": false, + "affects_scoring_rule": false, + "affects_metric_values": false, + "affects_architecture_decision": false, + "affects_execution_authorization": true, + "affects_reproducibility_surface": false, + "narrowing": "A61 REPAIRS THE GENERATOR THAT MINTS A CONTINUATION AUTHORIZATION, and changes no threshold, selection rule, metric definition, route derivation or architecture rule. score() is untouched and affects_metric_values is FALSE. No file joins or leaves METHODOLOGY_SURFACE, RESULT_BEARING_DATA or AUTHORIZATION_EXTRAS, so the authorization manifest is the same 34 entries before and after and affects_reproducibility_surface is FALSE; probes/x04_freeze_check.py is declared here because F9 protects every .py under the study directory, not because it is on that surface. THE DEADLOCK: the gate retires an authorization whose pinned DEVIATIONS.md blob has moved, while the generator tested only manifest_divergence over the methodology surface, which does not include the register. Declaring the operator's attestation in A57 moved the register without touching any surface file, so the gate read EXECUTION FORBIDDEN and the generator read 'nothing has changed'. The apparatus could not be advanced by any supported command. The generator now delegates that verdict to continuation_auth_errors, the gate's own validator, because two implementations of 'is this authorization still in force' is the defect and the missing register check was only where it surfaced. ONE SNAPSHOT PER RECORD: both artifact producers pinned blobs read from the working tree into records naming a HEAD commit, and nothing reconciled the two. The register reached neither check that would -- F9 exempts it by F9_IGNORE and F10 does not carry it -- so an uncommitted declaration could be pinned into a record claiming a head at which those bytes never existed, refused by historical validation only after the entry was already in an append-only chain. Every pin now resolves at the commit the record names, in the continuation producer AND in the execution marker, which carried the identical defect. build_execution_marker was extracted because its payload was built inline in main, where no control could reach it. ALSO: publication is now atomic and no-clobber via os.link, because a temp-and-rename overwrites and an exclusive create can strand a truncated record; the freeze and readiness gates are revalidated immediately before publication rather than only at entry; and the PERMITTED AS CONTINUATION line now names the chain entry actually in force, having read sequence 1 unconditionally. WHAT IS NOT CLOSED: there is no single atomic check before publication. Each read opens its own window and the last closes only at the link. A change that PERSISTS is retired by the gate on its next run; a TRANSIENT one, reverted before anything looks again, is not. This is a single-operator local repository, which is why that exposure is tolerable rather than answered. VERIFIED: the gate's own self-test at 270/270, and forty-five mutations each required to kill the control written for it. Several controls were found vacuous first and repaired -- a truthiness check under a comment claiming exactness, a substring match where 8 matched 18, an assertion comparing HEAD against HEAD, a refusal branch the fixture could never enter, and an ordering fix whose own undoing nothing detected. NOT ADDRESSED and tracked separately: G2's baseline fixture is already invalid, so several G2 controls pass for reasons unrelated to their labels. That predates this work and reworking those fixtures is its own change with its own review.", + "files_touched": [ + "probes/x04_freeze_check.py" + ] +} +``` + +## A62 — POST-BOUNDARY APPARATUS DEVIATION + +```json +{ + "id": "A62", + "kind": "DEVIATION", + "commits": [ + "4599505d" + ], + "classification": "POST-BOUNDARY APPARATUS DEVIATION (R1 ROUTE COMPARISON)", + "made_after_boundary": "de60dddf906bc4b01e5ffbe9af4d3e833a9a2be7 (continuation boundary)", + "results_already_visible": { + "members": 17, + "pages": 4190, + "d_frame_census": 13992, + "s1_documents_firing": "17/17", + "p_head_documents": 12, + "p_head_pages": 2864, + "cross_engine": "17/17 measured, n_qualified 0", + "ai_route": "122/122 adjudicated, committed at 0a6cdd02", + "oracle": "materialized at 8f9831a8" + }, + "affects_membership": false, + "affects_scoring_rule": false, + "affects_metric_values": true, + "affects_architecture_decision": true, + "affects_execution_authorization": false, + "affects_reproducibility_surface": true, + "narrowing": "A62 MAKES THE R1 ROUTE COMPARISON ASK THE A54 OWNER, and changes no threshold, selection rule, metric definition, frame derivation or architecture rule. A48 conditioned the frame-derived REQUIREMENT on the A27.3 budget but left `_required_r1_routes` reading the repeat's stored `adjudication_routes` raw, so the two sides of that comparison were derived under different rules. On the frozen pre-A48 key, whose routes came from RAW D membership, they can never agree: 1399 of 1401 R1 repeats mismatch, 1395 D-only declaring ['human'] against a required (), and 4 C+D declaring ['ai','human'] against ('ai',). `score_canonical()` refused outright. `score_metrics` was the ONLY route-requirement consumer outside `effective_record_routes`; `build_oracle.validate_adjudicated` and `ai_adjudicate.ai_route_items` already ask it. THIS IS THE THIRD LAYER OF ONE DEFECT: the first A48 pass conditioned route derivation, A48.7 found `heading_metrics` still walking (D_FRAME, PURPOSE_D_DECISION), and this is the declared-side comparison. Each was latent until the layer above it was unblocked, because scoring could never run end to end on the real key: `validate_inputs` refused without the AI route, which exists only as of the confirmatory adjudication at 0a6cdd02. affects_scoring_rule is FALSE: A62 implements the already-frozen A27.3 rule as A48 and A54 installed it and introduces no rule of its own. affects_metric_values is TRUE on the same reading A54 recorded: the R1 path moves from REFUSAL to the effective population, so the metric block produced is not the one the pre-A62 apparatus would have produced. affects_architecture_decision is TRUE at one remove: the repair makes the decision artifact REACHABLE, and the R1 verdict it yields sets A48-dependent `decided_by` attribution, though the outcome enum at a census above the budget remains constrained to INSUFFICIENT_COMPARATIVE_EVIDENCE and Rule 1 still cannot select corrected extended glyph. affects_reproducibility_surface is TRUE: `probes/score_metrics.py` is a member of the authorization manifest, so the continuation authorization at sequence 7 no longer speaks for the current apparatus. `probes/x27_score_metrics.py` is declared here because F9 protects every .py under the study directory, not because it is on that surface. THE REFUSAL IS UNWEAKENED: `reinterpret` is False for every key but the one pinned by a SHA-256 over its complete canonical content, and with it False `effective_record_routes` returns the stored routes unchanged, so a shortened repeat still mismatches its frame requirement and still refuses; the digest covers `adjudication_routes` and `human_answer_purposes` alike, so shortening the pinned key to earn the reinterpretation denies recognition instead. VERIFIED: x27 at 199/199, the gate's own self-test at 270/270, ruff clean. Every pre-existing R1 fixture is a post-A48 key for which reinterpretation never engages, so all 23 of them passed against an implementation that never enabled it at all; four checks now pin it in both directions and were shown RED before green. NOT ADDRESSED and tracked separately as issue 726: the scorer emits NOT_EVALUABLE_NO_R1_PAIRS for a zero-evidence R1 while `rule3_gates` accepts only NOT_EVALUABLE, so that verdict falls through to PASS. Not exercised by this run, since R1 text is FAIL on a non-empty denominator, and unrelated to route reinterpretation. ALSO NOT ADDRESSED, predating this work: G2's baseline fixture is already invalid, so several G2 controls pass for reasons unrelated to their labels.", + "files_touched": [ + "probes/score_metrics.py", + "probes/x27_score_metrics.py" + ] +} +``` + +### A62.1 — the R1 population collapse, and why no evidence floor was added + +**This is a finding about the study design under A27.3, not a defect and not a repair.** +It is recorded because the number it produces decides a Rule 3 gate. + +With the comparison fixed, R1 resolves to **6 pairs, all on the AI route, with a summed +heading-occurrence denominator of 5**: + +| dimension | observed | threshold | verdict | +|---|---|---|---| +| text | 4/5 = 0.80 | >= 0.90 | **FAIL** | +| role | 4/5 = 0.80 | >= 0.80 | **PASS**, by exact equality | + +Four pairs are one-vs-one matched and agree on both dimensions. One pair is one-sided: the +adjudicator enumerated no heading on the primary presentation and one on the repeat. Under +R6.1 that occurrence stays in the denominator and earns no numerator, so **that single item +alone produces both 4/5 figures**. The sixth pair is zero-vs-zero and contributes nothing, +which is why `n_pairs` reads 6 while the denominator is 5; the pair count overstates the +evidence and should not be read as the evidence base. + +**The AI arm was never a designed sample.** `plan_r1_repeats` draws an unstratified 10 % of +all eligible primaries, `floor(14016 * 0.10) = 1401`, with no frame or stratum quota. The D +frame outnumbers C by about 146:1, so a draw of 1395 D-only, 4 C+D and 2 C-only repeats is +an arithmetic consequence of the population shape rather than an allocation. R1's designed +power lived entirely in the human arm, and A27.3 as implemented by A48 removed that arm. +What remains is the residue of an unstratified draw, now carrying the whole gate. + +**What the five occurrences are.** Four carry the fine section 5.3 role `other` and are bill +designators (`†HR 6157 EAS`, `•S 1609 PCS`, `•HR 6147 RH`); one is `section` (`SEC. 504.`). +`other` belongs in R1: section 5.3's codebook includes it, section 5.6 restricts no role set, +and R6.3 records that M5 alone coarsens. Excluding designators would itself be post-hoc, and +the role is not causal in any case, since any one-sided occurrence fails both dimensions +identically. + +**No evidence floor was added, deliberately.** Section 5.6 freezes the 10 % repeat fraction +and the two thresholds and nothing else: no minimum pair count, no minimum denominator, no +stratification. A floor introduced now would take its value from having seen that a +denominator of 5 produces a FAIL, which is the substitution the pre-registration exists to +prevent. `_r1_status` therefore continues to treat every positive denominator as evaluable, +exactly as frozen. + +**The consequence is applied, not softened.** Section 5.6's R1 row reads "below -> text +metrics void / role metric void". The scorer reports that at `metrics.r1_text_gate`, which +names M2 and M3 as voided when the text dimension is FAIL; `decide_architecture` does not +read it, because Rule 3 (A27.6) owns the decision consequence and a failing Rule 3 blocker +already forecloses a comparative result. **So the artifact carries M2 and M3 flagged void on +the strength of five heading occurrences.** That is the conservative direction and it is what +section 5.6 froze: declining to apply it after seeing the number would be the softening the +register exists to refuse. + +**The contrary reading is recorded rather than dismissed.** A36.6's "C only -> AI, D only -> +human" and section 5.6's "10 % of regions presented twice" read literally keep the 1395 +D-only repeats as human-route items, in which case their absent answers would REFUSE rather +than resolve to no required route. A48 chose the other interpretation, that a +non-result-bearing route is absent and is not scored. That choice is what makes any metric +producible at all, and it is why every figure above is labelled NON-CONFIRMATORY under +section 4.7. + +**An auditability gap, reported because it is not closed.** `_r1_pair_facts` emits aggregate +counts per pair and not the occurrence keys or heading texts, so the scored artifact alone +cannot show WHICH item drove the FAIL or confirm that the four matched occurrences bind the +intended printed starts. Both were established out of band against the committed +adjudications for this entry and are not reconstructible from `metrics.json`. + +### A62.2 — the heading population is unsettled for running page furniture, and R1's verdict turns on it + +**This qualifies A62.1 and partly corrects it.** A62.1 recorded that four of R1's five +denominator-bearing occurrences are bill designators carrying the fine role `other`, and argued +the role was not causal because any one-sided occurrence fails both dimensions identically. That +is true of the role LABEL and beside the point about the POPULATION. Recomputed over the same +committed answers, with running page designators removed from both sides of every pair: + +| R1 text, AI route | numerator / denominator | ratio | verdict | +|---|---|---|---| +| as scored | 4 / 5 | 0.80 | **FAIL** | +| excluding running designators | 1 / 1 | 1.00 | **PASS** | + +The single occurrence that earns no numerator is `•S 1609 PCS`, reported on the repeat +presentation and not on the primary. It is a running page designator. Removing designators leaves +one occurrence, `SEC. 504.`, matched and agreeing on both text and role. **So the R1 text FAIL, +and with it section 5.6's void of M2 and M3, rests entirely on whether a running page designator +is a heading.** + +**The two routes answer that question differently, and the frozen prompt does not settle it.** + +| | headings reported | running page designators | roles | +|---|---|---|---| +| AI, 122 answers | 95 | **23** | section 32, account 23, other 35, agency 4, grouping 1 | +| human, 45 answers | 33 | **0** | section 9, UNREADABLE 23, agency 1 | + +Twenty-one designators carry a leading dagger or bullet (`†HR 6157 EAS`, `•S 1609 PCS`, +`•HR 6147 RH`, `•HR 7611 RH`, `•S 2976 PCS`, `•S 4663 IS`, `•HR 5961 RH`); two more are the same +running designator without the mark (`HR 7617 RFS`, `HR 8469 PCS`). The human reported none of +them across any answer. + +`adjudicator_prompt.md` defines a heading by printed composition alone: "centered, or set in +capitals, or set in italic, or set in a distinctly larger or heavier face, or otherwise +typographically separated from the paragraphs around it." Its exclusion list is "running body +text, ordinary paragraphs, tables of figures, dollar amounts, and the small numbers printed in +the left margin." A page-foot bill designator is set in bold capitals and is typographically +separated, and running page furniture is not excluded. **On a literal reading of the frozen +prompt the AI route is correct and the human route is applying an unstated exclusion.** Neither +is transcribing wrongly; the instrument does not say which is intended. + +This is a definitional gap in a frozen document, so it is recorded, not repaired. Deciding it now, +having seen that one answer yields R1 PASS and the other R1 FAIL, is the substitution +PRE-REGISTRATION 4.7 exists to prevent. What the run establishes is that the gap exists and that +it is load-bearing. + +**Reach beyond R1, stated as exposure rather than as a correction.** 23 of the 95 AI-route +headings are running designators, so they also sit in the M1 recall denominator and the M2/M3 +populations for the C estimand. No recomputation of those is offered here: doing so would be a +second post-hoc definition applied to a result already in hand. + +### A62.3 — the two adjudication routes fail differently, and both failures are systematic + +The N-A and N-B control failures recorded in the scoring commit are not scattered unreliability. +Both routes were examined against the rendered images they were shown. + +**The human route silently repairs spacing defects.** N-A outcomes by mutation variant: + +| variant | AI | human | +|---|---|---| +| DELETE_ONE_WORD | 3/3 | 3/3 | +| WELD_TWO_WORDS | 3/3 | **0/3** | +| SPLIT_ONE_WORD | 1/2 | **0/2** | + +Every deleted word is caught; every spacing mutation is normalised away. Three of the five +spacing fixtures carry no strike-through and render cleanly at 300 DPI in a serif face: +`OPERATIONSAND SUPPORT` is plainly legible and was transcribed `OPERATIONS AND SUPPORT`. This is +the failure class N-A exists to detect, and it is also the defect class the STUDY exists to +measure, so it bears directly on the human route's fitness as ground truth for weld and split +errors. + +**The human route substitutes a familiar spelling.** `RESCISSION` appears as `RECISSION` in 3 of +4 human headings and twice more in human free-text notes; the AI route has 4 of 4 correct and no +substitutions. Both N-B failures are this substitution. The images unambiguously print +`RESCISSION`. Both behaviours are reading-for-meaning: normalise the spacing, correct the word. + +**One N-A fixture is confounded and is the AI route's only N-A failure.** A SPLIT was applied to a +heading already carrying strike-through markup, and the strike line bridges the inserted gap. Both +routes read one word, and the AI's committed note reasons explicitly that the gap is a strike +artifact. The fixture's `mutation_evidence` validates the text layer only and asserts nothing +about legibility in the render. Tracked as issue 727 and NOT repaired here: `control_fixtures.json` +is on the result-bearing manifest, so editing it would retire the standing authorization and +invalidate the scored metrics, and rebuilding an instrument after seeing which result it produced +is the same post-hoc substitution refused above. Setting that fixture aside, the AI route is 8/8 +on N-A and 8/8 on N-B. + +**What the C-audit does and does not corroborate.** The 25 C-audit stimuli carry both routes' +answers; 16 of 25 agree exactly on text and role. Neither control behaviour recurs there, and that +is expected rather than reassuring: the C-audit contains no injected spacing defects to normalise +and few instances of the substituted word, so it cannot detect either. Of the 9 differences, 4 are +the designator disagreement of A62.2, 2 are role-only (the human marking `UNREADABLE` on text both +routes transcribed identically), and 3 are human transcription or segmentation errors confirmed +against the images: `SEC. 214.` read as `SEC. 215.` and merged with the following heading, an +omitted `SEC. 722.`, and a stray quotation mark with trailing space. On every one of the three the +AI route matches the image. + +**Direction, not rate.** One human across 8 N-A fixtures, 8 N-B fixtures and 25 C-audit stimuli. +The mechanisms are established; the frequencies are not. + +### A62.4 — the running-designator question cannot be settled on the frozen text, and R1's failure is re-attributed + +**This resolves the question A62.2 left open by establishing that it is not resolvable on the +instrument, and it corrects A62.2's account of what the R1 failure is evidence of.** Nothing here +changes a threshold, a metric definition, a selection rule or any committed value, and nothing is +recomputed. + +**A62.2's conclusion is confirmed, on evidence A62.2 did not have. The frozen prompt does not +settle it.** + +The reason is not that the definition is vague. It is that **the compositional test over-generates, +and every reader silently reads it down.** The first property in the definition's list is +"centered". A centered page folio is centered, is typographically separated from the text block, +and is not one of "the small numbers printed in the left margin", which are the marginal line +numbers. On a strict compositional reading a page number is therefore a heading. + +**No one read it that way.** The AI route's notes discuss a page folio in **25** answers and report +one as a heading in **none**. Across both routes, over all 167 committed answers, a bare-number +heading is reported **zero** times. Two of those folio exclusions sit in the R1 pair that agrees: +`9bf5371acb85cb21` and `5b3e82d843a78143` each report `SEC. 504.` and each explicitly set aside a +centered folio, one of them the numeral `504` itself. + +So the compositional test is universally applied subject to an unstated page-furniture carve-out. +**The frozen text nowhere says where that carve-out's boundary runs, and that boundary is exactly +the question.** It excludes folios on everyone's reading. Whether it also excludes a page-foot bill +designator is what the instrument does not say. + +This also disposes of the strongest argument for the literal reading, which is worth recording +because it is the one a reader reaches for first: that the exclusion list names left-margin numbers +and so shows the drafter had page furniture in view and enumerated it deliberately. The list +demonstrably fails to cover furniture that every reader excluded regardless. It is an incomplete +enumeration, not a considered one, and it cannot bear the weight of an expressio unius inference. + +**Two further readings were tested and neither settles it.** "Judge this visually, from composition +alone" is the prompt's own instruction to prefer composition over function, and it is the best +argument that a designator is a heading; but it is the same instruction that would admit folios, so +it cannot be applied strictly and the question of how far to read it down returns unanswered. +Against it, a structural presupposition runs through the fields surrounding the definition: every +`role` label but `other` names a structural element of an appropriations bill, and `parent` asks +for "the nearest heading above this one that **this heading sits under**". `other` does not resolve +this, because it classifies something already determined to be a heading rather than conferring +heading-hood on anything typographically distinct. + +**THE INSTRUMENT FINDING.** `adjudicator_prompt.md` defines a heading compositionally, surrounds it +with fields that presuppose structure, and states no rule for page furniture beyond one incomplete +example. A successor instrument should settle running page furniture in one explicit sentence +rather than leave it to be inferred from a composition test that, read strictly, admits page +numbers. + +**THE CORRECTION: the R1 text FAIL is not a disagreement between routes.** + +A62.2 presents the gap as AI route against human route and infers that the definitional answer +decides whether the text metrics stand. **A62.2's arithmetic sensitivity is not disputed here**: +filtering designators does change R1 from 0.80 to 1.00. What does not hold is the attribution. + +**Every R1 pair is on the AI route.** `metrics.json` `r1_reliability` carries `n_pairs` 6 and +`route` reads `ai` on all six. The human route contributes no R1 pair, having been removed from R1 +by A48, and carries no record at all for either id in the failing pair. **The human route is not +party to this failure.** + +**Both presentations print the designator, and the adjudicator saw it both times.** The pair is +primary `265fdcf236211f36`, repeat `8f6286a669bc6f38`. Section 5.6 specifies an R1 repeat as the +same region "re-rendered at a different but visually equivalent scale so they cannot be recognised +by hash", and that is what these are: 1499x1208 against 1649x1328, a uniform 1.100 scale, mean +absolute greyscale difference 2.6/255 after resampling to a common size, ink fraction 0.03246 +against 0.03273 overall and 0.04252 against 0.04311 in the foot band. Both images print +`•S 1609 PCS` in bold at the foot, confirmed by inspecting the rendered images rather than the text +layer. + +**The primary's committed note excludes it in terms:** "A bold page-foot line reading '•S 1609 PCS' +appears at the bottom... **It is set in bold and separated from the text block, but it is a page +footer/document identifier rather than a heading over any content**, so it is not reported as a +heading." The repeat reports the same string with role `other`. **This is not a perceptual miss. It +is one adjudicator applying two incompatible rules to the same object across two presentations of +the same region**, which is what section 5.6 says R1 exists to detect: "R1 exists because phase 1 +found six identical stimuli answered 3 BOUNDARY / 3 NO_BOUNDARY and reported it as a defect in its +own work." + +**The instability is specific to this class, which is the sharpest thing the run establishes.** + +| page furniture class | admitted as a heading | excluded | consistent? | +|---|---|---|---| +| centered page folio | 0 | 25 | **yes** | +| page-foot bill designator | 23 | 2 | **no** | + +The adjudicator holds a stable rule for folios and an unstable one for designators. The second +designator exclusion, `16f182372a39af6e` (`†HR 3237 EAS`, note: "page furniture (a running +footer/printer's mark), not a heading over any content"), sits in no R1 pair and moves no metric. +Every image whose last printed line carries the designator signature was accounted for: 12 +reported, 2 reasoned away, none overlooked. + +**THE CONSEQUENCE, stated without inventing a rule the protocol does not have.** + +| reading | R1 text | protocol status | +|---|---|---| +| as scored, designators counted | 4/5 = 0.80 | **FAIL**; section 5.6 voids M2 and M3 | +| designators excluded | 1/1 = 1.00 | **PASS**, and a genuine one | + +**The 1/1 is a real protocol PASS and is not dismissed here.** Section 5.6 sets no minimum pair +count and no minimum denominator, and A62.1 deliberately declined to add one after seeing that a +denominator of 5 produces a FAIL. Declaring 1/1 unevaluable now would install exactly that floor, +retrospectively, which is the substitution the register has already refused once. **It is a PASS on +one observation: very weak evidence, and NON-CONFIRMATORY under 4.7 like every other figure in this +run.** That is a statement about evidential weight, not about protocol status, and the two are kept +apart deliberately. + +**But the exclusion reading is not the cheap rescue A62.2's two-row table implies, and three costs +go unrecorded there.** First, excluding designators removes **4 of the 5 denominator-bearing +occurrences, 3 agreeing and the 1 failing**. Second, R6.1 scores against a symmetric union +denominator, so ruling designators out does not simply delete the repeat's report: **it converts +that report into an over-trigger**, and reaching 1/1 requires a further, unstated filter on the +output population as well as on the reference. Third, the same ruling would reclassify **19 of the +65 C-frame headings** as reported-but-not-headings, which bears on M1 precision and on what N-C +exists to detect. None of that has been computed. + +**So the definitional answer does not, by itself, decide whether the text metrics stand.** Under +the population as scored, R1 fails and section 5.6's void applies. Under the contrary reading, R1 +passes on a single observation while the route acquires an unexamined over-reporting problem +elsewhere. Neither outcome is a confirmatory result about reliability. + +**THE EXPOSURE, confirmed and resolved by population.** A62.2 gave 23 of 95, which is the count +across all 122 AI-route answers, and said those designators also sit in the M1 recall denominator. +Resolved against the key's frame flags: + +| population | headings | of which running designators | +|---|---|---| +| C frame | 65 | **19** | +| controls | 25 | 0 | +| R1 repeats | 5 | 4 | +| all AI answers | 95 | 23 | + +The C-frame total of 65 is exactly M1 recall's reported denominator, which corroborates the +mapping. **Roughly three in ten of the headings in the C estimand's adjudicated enumeration are +running page furniture whose status the instrument does not settle.** That is the exposure, and it +is larger for the C estimand than the 23-of-95 figure suggests. + +**WHAT STANDS.** The metrics are left exactly as scored, because that is what the committed +apparatus produced and because nothing in this entry authorizes a change: R1 text 4/5 FAIL, M2 and +M3 void under section 5.6, and the 19 C-frame designators in place. Applying section 5.6's frozen +consequence is the conservative direction and the one A62.1 already committed to; declining to +apply it now, having seen that the other reading would avoid it, is the softening the register +exists to refuse. + +**On PRE-REGISTRATION 4.7.** This question was examined in full knowledge of which answer yields +PASS and which yields FAIL. The outcome is that the instrument does not decide, which moves no +value and is the only outcome that does not substitute a post-hoc definition for a frozen one. The +exposure is stated rather than argued away: the examination happened after the results were +visible, and every figure it touches was already NON-CONFIRMATORY on other grounds. diff --git a/docs/research/pdf-backend-bakeoff/validation/external-validity/results/metrics.json b/docs/research/pdf-backend-bakeoff/validation/external-validity/results/metrics.json new file mode 100644 index 00000000..690890ff --- /dev/null +++ b/docs/research/pdf-backend-bakeoff/validation/external-validity/results/metrics.json @@ -0,0 +1,7477 @@ +{ + "adequacy_4_5": { + "kinds_counted": [ + "account", + "agency", + "grouping" + ], + "n_occurrences": 2583, + "owner": "decide_architecture consumes this (Rule 3); the verdict itself is a population fact", + "population_counted": "P-head", + "strata": [ + 1, + 2, + 3, + 4, + 6, + 7, + 8 + ], + "strata_filled": 7, + "verdict": "GENERALISABLE" + }, + "control_verdicts": { + "aggregation": "a kind PASSES only if every fixture passes on every required route; no tolerance", + "by_kind": { + "N-A": { + "by_route": { + "ai": { + "n_passed": 7, + "n_total": 8 + }, + "human": { + "n_passed": 3, + "n_total": 8 + } + }, + "failures": [ + { + "blind_id": "4c4e9f956425e6e8", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "OPERATIONSAND SUPPORT" + ], + "observed_texts": [ + "FEDERAL EMERGENCY MANAGEMENT AGENCY", + "OPERATIONS AND SUPPORT" + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "505a08c6b1741186", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "VETERANSINSURANCE AND INDEMNITIES" + ], + "observed_texts": [ + "VETERANS INSURANCE AND INDEMNITIES" + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "770bcee3f716e767", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "OPERA TIONS AND SUPPORT" + ], + "observed_texts": [ + "RESEARCH, DEVELOPMENT, TRAINING, AND SERVICES", + "U.S. CITIZENSHIP AND IMMIGRATION SERVICES", + "OPERATIONS AND SUPPORT" + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "ab2f34a86416bf77", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "(RESCI SSION OF FUNDS)" + ], + "observed_texts": [ + "(RESCISSION OF FUNDS)", + "SEC. 126." + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "ai" + }, + { + "blind_id": "ab2f34a86416bf77", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "(RESCI SSION OF FUNDS)" + ], + "observed_texts": [ + "(RESCISSION OF FUNDS)", + "SEC. 126." + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "c95132ee630b831b", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "(RESCISSIONSOF FUNDS)" + ], + "observed_texts": [ + "(RECISSIONS OF FUNDS)" + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + } + ], + "n_passed": 10, + "n_total": 16, + "status": "FAIL" + }, + "N-B": { + "by_route": { + "ai": { + "n_passed": 8, + "n_total": 8 + }, + "human": { + "n_passed": 6, + "n_total": 8 + } + }, + "failures": [ + { + "blind_id": "1ff91fd3a5a13695", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(RESCISSION OF FUNDS)" + ], + "observed_texts": [ + "(RECISSION OF FUNDS)", + "SEC. 128." + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "b082f75a36248652", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(RESCISSIONS OF FUNDS)" + ], + "observed_texts": [ + "(RECISSIONS OF FUNDS)", + "SEC. 240." + ], + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + } + ], + "n_passed": 14, + "n_total": 16, + "status": "FAIL" + }, + "N-C": { + "by_route": { + "ai": { + "n_passed": 4, + "n_total": 4 + }, + "human": { + "n_passed": 4, + "n_total": 4 + } + }, + "failures": [], + "n_passed": 8, + "n_total": 8, + "status": "PASS" + } + }, + "comparison": "exact raw string equality; NO normalisation (R8)", + "decision_owner": "decide_architecture -- Rule 3's gate vector (A27.6) consumes these three statuses; they are the FACTS, not the decision", + "frozen_population": { + "N-A": 8, + "N-B": 8, + "N-C": 4 + }, + "n_controls": 20, + "per_control": [ + { + "blind_id": "0a3a56317a1f986e", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "0a3a56317a1f986e", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "1ff91fd3a5a13695", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(RESCISSION OF FUNDS)" + ], + "observed_texts": [ + "(RESCISSION OF FUNDS)", + "SEC. 128." + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "1ff91fd3a5a13695", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(RESCISSION OF FUNDS)" + ], + "observed_texts": [ + "(RECISSION OF FUNDS)", + "SEC. 128." + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "232e3a3bddc6048d", + "control_kind": "N-A", + "control_variant": "DELETE_ONE_WORD", + "expected": [ + "AND SUPPORT" + ], + "observed_texts": [ + "COUNTERING WEAPONS OF MASS DESTRUCTION OFFICE", + "AND SUPPORT" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "232e3a3bddc6048d", + "control_kind": "N-A", + "control_variant": "DELETE_ONE_WORD", + "expected": [ + "AND SUPPORT" + ], + "observed_texts": [ + "COUNTERING WEAPONS OF MASS DESTRUCTION OFFICE", + "AND SUPPORT" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "4c2b1e375068050c", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "VETERANS HOUSING BENEFIT PROGRAM FUND" + ], + "observed_texts": [ + "VETERANS HOUSING BENEFIT PROGRAM FUND" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "4c2b1e375068050c", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "VETERANS HOUSING BENEFIT PROGRAM FUND" + ], + "observed_texts": [ + "VETERANS HOUSING BENEFIT PROGRAM FUND" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "4c4e9f956425e6e8", + "control_kind": "N-A", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "OPERATIONSAND SUPPORT" + ], + "observed_texts": [ + "FEDERAL EMERGENCY MANAGEMENT AGENCY", + "OPERATIONSAND SUPPORT" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "4c4e9f956425e6e8", + "control_kind": "N-A", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "OPERATIONSAND SUPPORT" + ], + "observed_texts": [ + "FEDERAL EMERGENCY MANAGEMENT AGENCY", + "OPERATIONS AND SUPPORT" + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "505a08c6b1741186", + "control_kind": "N-A", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "VETERANSINSURANCE AND INDEMNITIES" + ], + "observed_texts": [ + "VETERANSINSURANCE AND INDEMNITIES" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "505a08c6b1741186", + "control_kind": "N-A", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "VETERANSINSURANCE AND INDEMNITIES" + ], + "observed_texts": [ + "VETERANS INSURANCE AND INDEMNITIES" + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "6aa6a55bcbdd7c31", + "control_kind": "N-A", + "control_variant": "DELETE_ONE_WORD", + "expected": [ + "AND PENSIONS" + ], + "observed_texts": [ + "AND PENSIONS", + "(INCLUDING TRANSFER OF FUNDS)" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "6aa6a55bcbdd7c31", + "control_kind": "N-A", + "control_variant": "DELETE_ONE_WORD", + "expected": [ + "AND PENSIONS" + ], + "observed_texts": [ + "AND PENSIONS", + "(INCLUDING TRANSFER OF FUNDS)" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "6eacf09d8a3bbc51", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "6eacf09d8a3bbc51", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "770bcee3f716e767", + "control_kind": "N-A", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "OPERA TIONS AND SUPPORT" + ], + "observed_texts": [ + "RESEARCH, DEVELOPMENT, TRAINING, AND SERVICES", + "U.S. Citizenship and Immigration Services", + "OPERA TIONS AND SUPPORT" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "770bcee3f716e767", + "control_kind": "N-A", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "OPERA TIONS AND SUPPORT" + ], + "observed_texts": [ + "RESEARCH, DEVELOPMENT, TRAINING, AND SERVICES", + "U.S. CITIZENSHIP AND IMMIGRATION SERVICES", + "OPERATIONS AND SUPPORT" + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "780c70da198f7c4e", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "observed_texts": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "780c70da198f7c4e", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "observed_texts": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "86a2c7f959165893", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "FOREIGN CURRENCY FLUCTUATIONS ACCOUNT" + ], + "observed_texts": [ + "FOREIGN CURRENCY FLUCTUATIONS ACCOUNT" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "86a2c7f959165893", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "FOREIGN CURRENCY FLUCTUATIONS ACCOUNT" + ], + "observed_texts": [ + "FOREIGN CURRENCY FLUCTUATIONS ACCOUNT" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "94c109da1d772c62", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "94c109da1d772c62", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "9a3cbd5f467d70bc", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "9a3cbd5f467d70bc", + "control_kind": "N-C", + "control_variant": "SYNTHETIC_HEADING_FREE", + "expected": [], + "observed_texts": [], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "ab2f34a86416bf77", + "control_kind": "N-A", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "(RESCI SSION OF FUNDS)" + ], + "observed_texts": [ + "(RESCISSION OF FUNDS)", + "SEC. 126." + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "ai" + }, + { + "blind_id": "ab2f34a86416bf77", + "control_kind": "N-A", + "control_variant": "SPLIT_ONE_WORD", + "expected": [ + "(RESCI SSION OF FUNDS)" + ], + "observed_texts": [ + "(RESCISSION OF FUNDS)", + "SEC. 126." + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "b082f75a36248652", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(RESCISSIONS OF FUNDS)" + ], + "observed_texts": [ + "(RESCISSIONS OF FUNDS)", + "SEC. 240." + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "b082f75a36248652", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(RESCISSIONS OF FUNDS)" + ], + "observed_texts": [ + "(RECISSIONS OF FUNDS)", + "SEC. 240." + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "b4243941ae0a18de", + "control_kind": "N-A", + "control_variant": "DELETE_ONE_WORD", + "expected": [ + "RESEARCH AND" + ], + "observed_texts": [ + "RESEARCH AND" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "b4243941ae0a18de", + "control_kind": "N-A", + "control_variant": "DELETE_ONE_WORD", + "expected": [ + "RESEARCH AND" + ], + "observed_texts": [ + "RESEARCH AND" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "c7c3f56ec0ea15c7", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(INCLUDING TRANSFER OF FUNDS)" + ], + "observed_texts": [ + "(INCLUDING TRANSFER OF FUNDS)" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "c7c3f56ec0ea15c7", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "(INCLUDING TRANSFER OF FUNDS)" + ], + "observed_texts": [ + "(INCLUDING TRANSFER OF FUNDS)" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "c95132ee630b831b", + "control_kind": "N-A", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "(RESCISSIONSOF FUNDS)" + ], + "observed_texts": [ + "(RESCISSIONSOF FUNDS)", + "SEC. 237." + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "c95132ee630b831b", + "control_kind": "N-A", + "control_variant": "WELD_TWO_WORDS", + "expected": [ + "(RESCISSIONSOF FUNDS)" + ], + "observed_texts": [ + "(RECISSIONS OF FUNDS)" + ], + "pass": false, + "reason": "EXPECTED_TARGET_ABSENT", + "route": "human" + }, + { + "blind_id": "cdbfbcc59494ef86", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "NATIONAL FLOOD INSURANCE FUND" + ], + "observed_texts": [ + "NATIONAL FLOOD INSURANCE FUND" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "cdbfbcc59494ef86", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "NATIONAL FLOOD INSURANCE FUND" + ], + "observed_texts": [ + "NATIONAL FLOOD INSURANCE FUND" + ], + "pass": true, + "reason": null, + "route": "human" + }, + { + "blind_id": "f51516cbfd46896e", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "observed_texts": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "pass": true, + "reason": null, + "route": "ai" + }, + { + "blind_id": "f51516cbfd46896e", + "control_kind": "N-B", + "control_variant": "XML_CORROBORATED_HEADING", + "expected": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "observed_texts": [ + "PROCUREMENT, CONSTRUCTION, AND IMPROVEMENTS" + ], + "pass": true, + "reason": null, + "route": "human" + } + ], + "population_present": true, + "required_routes": [ + "ai", + "human" + ], + "ruled_by": "A41.2 R8" + }, + "cross_engine_qualification": { + "both_headlines_qualified": false, + "confirmatory_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "decision_blocking": false, + "failed_documents": [], + "headline_qualifications": { + "RQ1": null, + "RQ2": null + }, + "n_documents": 17, + "n_failed": 0, + "per_document": { + "113-hr-933": null, + "114-s-3001": null, + "115-hr-5961": null, + "115-hr-6147": null, + "115-hr-6157": null, + "115-s-1609": null, + "115-s-2976": null, + "116-hr-7611": null, + "116-hr-7617": null, + "117-hr-3237": null, + "117-s-4663": null, + "119-hr-6938": null, + "119-hr-7148": null, + "119-hr-8469": null, + "CRPT-114HRPT215": null, + "CRPT-114HRPT605": null, + "CRPT-119HRPT632": null + }, + "threshold_owner": "x09_skeleton_cross_engine.gate (document 0.95 / page 0.75)" + }, + "decision_owner": "decide_architecture (section 7) -- no Rule 0/1/3 outcome appears here", + "decision_taken_here": false, + "documents_scored": [ + "113-hr-933", + "114-s-3001", + "115-hr-5961", + "115-hr-6147", + "115-hr-6157", + "115-s-1609", + "115-s-2976", + "116-hr-7611", + "116-hr-7617", + "117-hr-3237", + "117-s-4663", + "119-hr-6938", + "119-hr-7148", + "119-hr-8469", + "CRPT-114HRPT215", + "CRPT-114HRPT605", + "CRPT-119HRPT632" + ], + "heading_population_notes": { + "estimand_purposes": [ + "c_metrics", + "d_decision", + "c_audit" + ], + "excluded_stimuli": { + "control": 20, + "r1_repeat": 1401 + }, + "pooling_rule": "C and D are separate estimands (A36.3); they are never summed together", + "routing": { + "c_metrics": "ai" + } + }, + "headings_pooled": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 42, + "metric": "M1 precision", + "numerator": 38, + "status": "REPORTED", + "value": 0.9047619047619048 + }, + "recall": { + "denominator": 65, + "metric": "M1 recall", + "numerator": 38, + "status": "REPORTED", + "value": 0.5846153846153846 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 65, + "metric": "M1 recall (SECONDARY)", + "numerator": 38, + "status": "REPORTED", + "value": 0.5846153846153846 + } + }, + "X": { + "precision": { + "denominator": 42, + "metric": "M1 precision", + "numerator": 38, + "status": "REPORTED", + "value": 0.9047619047619048 + }, + "recall": { + "denominator": 65, + "metric": "M1 recall", + "numerator": 38, + "status": "REPORTED", + "value": 0.5846153846153846 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 65, + "metric": "M1 recall (SECONDARY)", + "numerator": 38, + "status": "REPORTED", + "value": 0.5846153846153846 + } + } + }, + "M2": { + "H": { + "denominator": 38, + "metric": "M2", + "numerator": 10, + "status": "REPORTED", + "value": 0.2631578947368421 + }, + "X": { + "denominator": 38, + "metric": "M2", + "numerator": 10, + "status": "REPORTED", + "value": 0.2631578947368421 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 530, + "SPLIT": 0, + "TEXT_ERROR": 445, + "WELD": 0 + }, + "X": { + "OK": 530, + "SPLIT": 0, + "TEXT_ERROR": 445, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 65, + "metric": "M3", + "numerator": 10, + "status": "REPORTED", + "value": 0.15384615384615385 + }, + "X": { + "denominator": 65, + "metric": "M3", + "numerator": 10, + "status": "REPORTED", + "value": 0.15384615384615385 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 10, + "BOTH_DIRTY": 55, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 3, + "metric": "M4", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "X": { + "denominator": 3, + "metric": "M4", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "excluded_off_region": 41, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 36, + "metric": "M5", + "numerator": 35, + "status": "REPORTED", + "value": 0.9722222222222222 + }, + "excluded_unscorable": 2 + }, + "X": { + "agreement": { + "denominator": 36, + "metric": "M5", + "numerator": 35, + "status": "REPORTED", + "value": 0.9722222222222222 + }, + "excluded_unscorable": 2 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 10, + "X": 10 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 38, + "X": 38 + }, + "m3_boundary": { + "H": { + "OK": 530, + "SPLIT": 0, + "TEXT_ERROR": 445, + "WELD": 0 + }, + "X": { + "OK": 530, + "SPLIT": 0, + "TEXT_ERROR": 445, + "WELD": 0 + } + }, + "m3_clean": { + "H": 10, + "X": 10 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 10, + "BOTH_DIRTY": 55, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 65, + "X": 65 + }, + "m4_correct": { + "H": 2, + "X": 2 + }, + "m4_excluded_off_region": 41, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 3, + "X": 3 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 35, + "X": 35 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 2, + "X": 2 + }, + "m5_scored": { + "H": 36, + "X": 36 + }, + "n_adjudicated": 65, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 42, + "X": 42 + }, + "n_matched": { + "H": 38, + "X": 38 + }, + "n_stimuli": 96 + }, + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "owns": "PRE-REGISTRATION section 6 metrics M0, M1, M2, M3, M4, M5, M7 and M9; section 8; HARNESS-PLAN section 5", + "per_document": { + "113-hr-933": { + "M0": { + "M0_any": 3310, + "M0_any_rate": { + "denominator": 14746, + "metric": "M0-any", + "numerator": 3310, + "status": "REPORTED", + "value": 0.22446765224467652 + }, + "M0_any_rate_ALL_LINES_superseded": 0.216072, + "M0a_only_text": 3310, + "M0a_text": 3310, + "M0a_text_rate": { + "denominator": 14746, + "metric": "M0a", + "numerator": 3310, + "status": "REPORTED", + "value": 0.22446765224467652 + }, + "M0b_defined": 14746, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 14746, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 14746, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 367, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 2285, + "metric": "M0c", + "numerator": 367, + "status": "REPORTED", + "value": 0.16061269146608315 + }, + "both_absent": 573, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 15319, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 14746, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 1097, + "rate": { + "denominator": 1097, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 1097, + "rate": { + "denominator": 1097, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 11468, + "n_margin_numbered_lines": 10825, + "n_margin_numbered_with_glyph_size": 10825, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 14745, + "n_margin_numbered_lines": 14098, + "n_margin_numbered_with_glyph_size": 14098, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "113-hr-933", + "margin_line_loss": { + "deficit": 3273, + "fires": true, + "h": 10825, + "loser": "H", + "x": 14098 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 2, + "metric": "M1 precision", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 3, + "metric": "M1 recall", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 3, + "metric": "M1 recall (SECONDARY)", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + } + }, + "X": { + "precision": { + "denominator": 2, + "metric": "M1 precision", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 3, + "metric": "M1 recall", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 3, + "metric": "M1 recall (SECONDARY)", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + } + } + }, + "M2": { + "H": { + "denominator": 2, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 2, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 13, + "SPLIT": 0, + "TEXT_ERROR": 11, + "WELD": 0 + }, + "X": { + "OK": 13, + "SPLIT": 0, + "TEXT_ERROR": 11, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 3, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 3, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 3, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 2, + "metric": "M5", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 2, + "metric": "M5", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 0, + "X": 0 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 2, + "X": 2 + }, + "m3_boundary": { + "H": { + "OK": 13, + "SPLIT": 0, + "TEXT_ERROR": 11, + "WELD": 0 + }, + "X": { + "OK": 13, + "SPLIT": 0, + "TEXT_ERROR": 11, + "WELD": 0 + } + }, + "m3_clean": { + "H": 0, + "X": 0 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 3, + "X": 3 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 3, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 2, + "X": 2 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 2, + "X": 2 + }, + "n_adjudicated": 3, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 2, + "X": 2 + }, + "n_matched": { + "H": 2, + "X": 2 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 8 + }, + "114-s-3001": { + "M0": { + "M0_any": 410, + "M0_any_rate": { + "denominator": 1845, + "metric": "M0-any", + "numerator": 410, + "status": "REPORTED", + "value": 0.2222222222222222 + }, + "M0_any_rate_ALL_LINES_superseded": 0.205308, + "M0a_only_text": 410, + "M0a_text": 410, + "M0a_text_rate": { + "denominator": 1845, + "metric": "M0a", + "numerator": 410, + "status": "REPORTED", + "value": 0.2222222222222222 + }, + "M0b_defined": 1845, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 1845, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 1845, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 296, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 152, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 1997, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 1845, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 136, + "rate": { + "denominator": 136, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 136, + "rate": { + "denominator": 136, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1437, + "n_margin_numbered_lines": 1420, + "n_margin_numbered_with_glyph_size": 1420, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1845, + "n_margin_numbered_lines": 1826, + "n_margin_numbered_with_glyph_size": 1826, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "114-s-3001", + "margin_line_loss": { + "deficit": 406, + "fires": true, + "h": 1420, + "loser": "H", + "x": 1826 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 3, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 3, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 3, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + } + }, + "X": { + "precision": { + "denominator": 3, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 3, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 3, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + } + } + }, + "M2": { + "H": { + "denominator": 3, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 3, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + }, + "X": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 3, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 3, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 3, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 3, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 3, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 0, + "X": 0 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 3, + "X": 3 + }, + "m3_boundary": { + "H": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + }, + "X": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + } + }, + "m3_clean": { + "H": 0, + "X": 0 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 3, + "X": 3 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 3, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 3, + "X": 3 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 3, + "X": 3 + }, + "n_adjudicated": 3, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 3, + "X": 3 + }, + "n_matched": { + "H": 3, + "X": 3 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 2 + }, + "115-hr-5961": { + "M0": { + "M0_any": 712, + "M0_any_rate": { + "denominator": 2905, + "metric": "M0-any", + "numerator": 712, + "status": "REPORTED", + "value": 0.2450946643717728 + }, + "M0_any_rate_ALL_LINES_superseded": 0.218338, + "M0a_only_text": 712, + "M0a_text": 712, + "M0a_text_rate": { + "denominator": 2905, + "metric": "M0a", + "numerator": 712, + "status": "REPORTED", + "value": 0.2450946643717728 + }, + "M0b_defined": 2905, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 2905, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 2905, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 471, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 356, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 3261, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 2905, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 206, + "rate": { + "denominator": 206, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 206, + "rate": { + "denominator": 206, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 2216, + "n_margin_numbered_lines": 2197, + "n_margin_numbered_with_glyph_size": 2197, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 2905, + "n_margin_numbered_lines": 2886, + "n_margin_numbered_with_glyph_size": 2886, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "115-hr-5961", + "margin_line_loss": { + "deficit": 689, + "fires": true, + "h": 2197, + "loser": "H", + "x": 2886 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 6, + "metric": "M1 precision", + "numerator": 6, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 6, + "status": "REPORTED", + "value": 0.8571428571428571 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 6, + "status": "REPORTED", + "value": 0.8571428571428571 + } + }, + "X": { + "precision": { + "denominator": 6, + "metric": "M1 precision", + "numerator": 6, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 6, + "status": "REPORTED", + "value": 0.8571428571428571 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 6, + "status": "REPORTED", + "value": 0.8571428571428571 + } + } + }, + "M2": { + "H": { + "denominator": 6, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 0.16666666666666666 + }, + "X": { + "denominator": 6, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 0.16666666666666666 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 67, + "SPLIT": 0, + "TEXT_ERROR": 14, + "WELD": 0 + }, + "X": { + "OK": 67, + "SPLIT": 0, + "TEXT_ERROR": 14, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 7, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.14285714285714285 + }, + "X": { + "denominator": 7, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.14285714285714285 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 6, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 6, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 6, + "metric": "M5", + "numerator": 6, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 6, + "metric": "M5", + "numerator": 6, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 1, + "X": 1 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 6, + "X": 6 + }, + "m3_boundary": { + "H": { + "OK": 67, + "SPLIT": 0, + "TEXT_ERROR": 14, + "WELD": 0 + }, + "X": { + "OK": 67, + "SPLIT": 0, + "TEXT_ERROR": 14, + "WELD": 0 + } + }, + "m3_clean": { + "H": 1, + "X": 1 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 6, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 7, + "X": 7 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 6, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 6, + "X": 6 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 6, + "X": 6 + }, + "n_adjudicated": 7, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 6, + "X": 6 + }, + "n_matched": { + "H": 6, + "X": 6 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 1 + }, + "115-hr-6147": { + "M0": { + "M0_any": 848, + "M0_any_rate": { + "denominator": 3588, + "metric": "M0-any", + "numerator": 848, + "status": "REPORTED", + "value": 0.23634336677814938 + }, + "M0_any_rate_ALL_LINES_superseded": 0.210631, + "M0a_only_text": 848, + "M0a_text": 848, + "M0a_text_rate": { + "denominator": 3588, + "metric": "M0a", + "numerator": 848, + "status": "REPORTED", + "value": 0.23634336677814938 + }, + "M0b_defined": 3588, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 3588, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 3588, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 584, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 438, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 4026, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 3588, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 293, + "rate": { + "denominator": 293, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 293, + "rate": { + "denominator": 293, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 2778, + "n_margin_numbered_lines": 2760, + "n_margin_numbered_with_glyph_size": 2760, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 3588, + "n_margin_numbered_lines": 3570, + "n_margin_numbered_with_glyph_size": 3570, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "115-hr-6147", + "margin_line_loss": { + "deficit": 810, + "fires": true, + "h": 2760, + "loser": "H", + "x": 3570 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 1, + "metric": "M1 precision", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 4, + "metric": "M1 recall", + "numerator": 1, + "status": "REPORTED", + "value": 0.25 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 4, + "metric": "M1 recall (SECONDARY)", + "numerator": 1, + "status": "REPORTED", + "value": 0.25 + } + }, + "X": { + "precision": { + "denominator": 1, + "metric": "M1 precision", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 4, + "metric": "M1 recall", + "numerator": 1, + "status": "REPORTED", + "value": 0.25 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 4, + "metric": "M1 recall (SECONDARY)", + "numerator": 1, + "status": "REPORTED", + "value": 0.25 + } + } + }, + "M2": { + "H": { + "denominator": 1, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "X": { + "denominator": 1, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 19, + "SPLIT": 0, + "TEXT_ERROR": 27, + "WELD": 0 + }, + "X": { + "OK": 19, + "SPLIT": 0, + "TEXT_ERROR": 27, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 4, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.25 + }, + "X": { + "denominator": 4, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.25 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 1, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 1, + "metric": "M5", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 1, + "metric": "M5", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 1, + "X": 1 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 1, + "X": 1 + }, + "m3_boundary": { + "H": { + "OK": 19, + "SPLIT": 0, + "TEXT_ERROR": 27, + "WELD": 0 + }, + "X": { + "OK": 19, + "SPLIT": 0, + "TEXT_ERROR": 27, + "WELD": 0 + } + }, + "m3_clean": { + "H": 1, + "X": 1 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 4, + "X": 4 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 1, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 1, + "X": 1 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 1, + "X": 1 + }, + "n_adjudicated": 4, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 1, + "X": 1 + }, + "n_matched": { + "H": 1, + "X": 1 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 1 + }, + "115-hr-6157": { + "M0": { + "M0_any": 1858, + "M0_any_rate": { + "denominator": 8593, + "metric": "M0-any", + "numerator": 1858, + "status": "REPORTED", + "value": 0.21622250669149307 + }, + "M0_any_rate_ALL_LINES_superseded": 0.208039, + "M0a_only_text": 1858, + "M0a_text": 1858, + "M0a_text_rate": { + "denominator": 8593, + "metric": "M0a", + "numerator": 1858, + "status": "REPORTED", + "value": 0.21622250669149307 + }, + "M0b_defined": 8593, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 8593, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 8593, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 1333, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 338, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 8931, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 8593, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 644, + "rate": { + "denominator": 644, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 644, + "rate": { + "denominator": 644, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 6751, + "n_margin_numbered_lines": 6403, + "n_margin_numbered_with_glyph_size": 6403, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 8593, + "n_margin_numbered_lines": 8243, + "n_margin_numbered_with_glyph_size": 8243, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "115-hr-6157", + "margin_line_loss": { + "deficit": 1840, + "fires": true, + "h": 6403, + "loser": "H", + "x": 8243 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 4, + "metric": "M1 precision", + "numerator": 4, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + } + }, + "X": { + "precision": { + "denominator": 4, + "metric": "M1 precision", + "numerator": 4, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + } + } + }, + "M2": { + "H": { + "denominator": 4, + "metric": "M2", + "numerator": 2, + "status": "REPORTED", + "value": 0.5 + }, + "X": { + "denominator": 4, + "metric": "M2", + "numerator": 2, + "status": "REPORTED", + "value": 0.5 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 76, + "SPLIT": 0, + "TEXT_ERROR": 32, + "WELD": 0 + }, + "X": { + "OK": 76, + "SPLIT": 0, + "TEXT_ERROR": 32, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 7, + "metric": "M3", + "numerator": 2, + "status": "REPORTED", + "value": 0.2857142857142857 + }, + "X": { + "denominator": 7, + "metric": "M3", + "numerator": 2, + "status": "REPORTED", + "value": 0.2857142857142857 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 2, + "BOTH_DIRTY": 5, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 4, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 4, + "metric": "M5", + "numerator": 4, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 4, + "metric": "M5", + "numerator": 4, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 2, + "X": 2 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 4, + "X": 4 + }, + "m3_boundary": { + "H": { + "OK": 76, + "SPLIT": 0, + "TEXT_ERROR": 32, + "WELD": 0 + }, + "X": { + "OK": 76, + "SPLIT": 0, + "TEXT_ERROR": 32, + "WELD": 0 + } + }, + "m3_clean": { + "H": 2, + "X": 2 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 2, + "BOTH_DIRTY": 5, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 7, + "X": 7 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 4, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 4, + "X": 4 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 4, + "X": 4 + }, + "n_adjudicated": 7, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 4, + "X": 4 + }, + "n_matched": { + "H": 4, + "X": 4 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 3 + }, + "115-s-1609": { + "M0": { + "M0_any": 376, + "M0_any_rate": { + "denominator": 1551, + "metric": "M0-any", + "numerator": 376, + "status": "REPORTED", + "value": 0.24242424242424243 + }, + "M0_any_rate_ALL_LINES_superseded": 0.22421, + "M0a_only_text": 376, + "M0a_text": 376, + "M0a_text_rate": { + "denominator": 1551, + "metric": "M0a", + "numerator": 376, + "status": "REPORTED", + "value": 0.24242424242424243 + }, + "M0b_defined": 1551, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 1551, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 1551, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 1, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 251, + "metric": "M0c", + "numerator": 1, + "status": "REPORTED", + "value": 0.00398406374501992 + }, + "both_absent": 126, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 1677, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 1551, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 129, + "rate": { + "denominator": 129, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 129, + "rate": { + "denominator": 129, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1178, + "n_margin_numbered_lines": 1162, + "n_margin_numbered_with_glyph_size": 1162, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1551, + "n_margin_numbered_lines": 1535, + "n_margin_numbered_with_glyph_size": 1535, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "115-s-1609", + "margin_line_loss": { + "deficit": 373, + "fires": true, + "h": 1162, + "loser": "H", + "x": 1535 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 3, + "metric": "M1 precision", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "recall": { + "denominator": 8, + "metric": "M1 recall", + "numerator": 2, + "status": "REPORTED", + "value": 0.25 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 8, + "metric": "M1 recall (SECONDARY)", + "numerator": 2, + "status": "REPORTED", + "value": 0.25 + } + }, + "X": { + "precision": { + "denominator": 3, + "metric": "M1 precision", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "recall": { + "denominator": 8, + "metric": "M1 recall", + "numerator": 2, + "status": "REPORTED", + "value": 0.25 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 8, + "metric": "M1 recall (SECONDARY)", + "numerator": 2, + "status": "REPORTED", + "value": 0.25 + } + } + }, + "M2": { + "H": { + "denominator": 2, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 0.5 + }, + "X": { + "denominator": 2, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 0.5 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 49, + "SPLIT": 0, + "TEXT_ERROR": 170, + "WELD": 0 + }, + "X": { + "OK": 49, + "SPLIT": 0, + "TEXT_ERROR": 170, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 8, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.125 + }, + "X": { + "denominator": 8, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.125 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 7, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 1, + "metric": "M4", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 1, + "metric": "M4", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "excluded_off_region": 3, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 2, + "metric": "M5", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 2, + "metric": "M5", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 1, + "X": 1 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 2, + "X": 2 + }, + "m3_boundary": { + "H": { + "OK": 49, + "SPLIT": 0, + "TEXT_ERROR": 170, + "WELD": 0 + }, + "X": { + "OK": 49, + "SPLIT": 0, + "TEXT_ERROR": 170, + "WELD": 0 + } + }, + "m3_clean": { + "H": 1, + "X": 1 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 7, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 8, + "X": 8 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 3, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 1, + "X": 1 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 2, + "X": 2 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 2, + "X": 2 + }, + "n_adjudicated": 8, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 3, + "X": 3 + }, + "n_matched": { + "H": 2, + "X": 2 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 2 + }, + "115-s-2976": { + "M0": { + "M0_any": 559, + "M0_any_rate": { + "denominator": 2449, + "metric": "M0-any", + "numerator": 559, + "status": "REPORTED", + "value": 0.2282564311964067 + }, + "M0_any_rate_ALL_LINES_superseded": 0.210864, + "M0a_only_text": 559, + "M0a_text": 559, + "M0a_text_rate": { + "denominator": 2449, + "metric": "M0a", + "numerator": 559, + "status": "REPORTED", + "value": 0.2282564311964067 + }, + "M0b_defined": 2449, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 2449, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 2449, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 391, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 202, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 2651, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 2449, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 181, + "rate": { + "denominator": 181, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 181, + "rate": { + "denominator": 181, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1890, + "n_margin_numbered_lines": 1872, + "n_margin_numbered_with_glyph_size": 1872, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 2449, + "n_margin_numbered_lines": 2431, + "n_margin_numbered_with_glyph_size": 2431, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "115-s-2976", + "margin_line_loss": { + "deficit": 559, + "fires": true, + "h": 1872, + "loser": "H", + "x": 2431 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 2, + "metric": "M1 precision", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 5, + "metric": "M1 recall", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 5, + "metric": "M1 recall (SECONDARY)", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + } + }, + "X": { + "precision": { + "denominator": 2, + "metric": "M1 precision", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 5, + "metric": "M1 recall", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 5, + "metric": "M1 recall (SECONDARY)", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + } + } + }, + "M2": { + "H": { + "denominator": 2, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 2, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 12, + "SPLIT": 0, + "TEXT_ERROR": 29, + "WELD": 0 + }, + "X": { + "OK": 12, + "SPLIT": 0, + "TEXT_ERROR": 29, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 5, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 5, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 5, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 2, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 2, + "metric": "M5", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 2, + "metric": "M5", + "numerator": 2, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 0, + "X": 0 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 2, + "X": 2 + }, + "m3_boundary": { + "H": { + "OK": 12, + "SPLIT": 0, + "TEXT_ERROR": 29, + "WELD": 0 + }, + "X": { + "OK": 12, + "SPLIT": 0, + "TEXT_ERROR": 29, + "WELD": 0 + } + }, + "m3_clean": { + "H": 0, + "X": 0 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 5, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 5, + "X": 5 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 2, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 2, + "X": 2 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 2, + "X": 2 + }, + "n_adjudicated": 5, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 2, + "X": 2 + }, + "n_matched": { + "H": 2, + "X": 2 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 2 + }, + "116-hr-7611": { + "M0": { + "M0_any": 225, + "M0_any_rate": { + "denominator": 1066, + "metric": "M0-any", + "numerator": 225, + "status": "REPORTED", + "value": 0.21106941838649157 + }, + "M0_any_rate_ALL_LINES_superseded": 0.1875, + "M0a_only_text": 225, + "M0a_text": 225, + "M0a_text_rate": { + "denominator": 1066, + "metric": "M0a", + "numerator": 225, + "status": "REPORTED", + "value": 0.21106941838649157 + }, + "M0b_defined": 1066, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 1066, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 1066, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 1, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 175, + "metric": "M0c", + "numerator": 1, + "status": "REPORTED", + "value": 0.005714285714285714 + }, + "both_absent": 134, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 1200, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 1066, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 111, + "rate": { + "denominator": 111, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 111, + "rate": { + "denominator": 111, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 852, + "n_margin_numbered_lines": 835, + "n_margin_numbered_with_glyph_size": 835, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1066, + "n_margin_numbered_lines": 1049, + "n_margin_numbered_with_glyph_size": 1049, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "116-hr-7611", + "margin_line_loss": { + "deficit": 214, + "fires": true, + "h": 835, + "loser": "H", + "x": 1049 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 4, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 0.75 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 0.42857142857142855 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 0.42857142857142855 + } + }, + "X": { + "precision": { + "denominator": 4, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 0.75 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 0.42857142857142855 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 0.42857142857142855 + } + } + }, + "M2": { + "H": { + "denominator": 3, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 0.3333333333333333 + }, + "X": { + "denominator": 3, + "metric": "M2", + "numerator": 1, + "status": "REPORTED", + "value": 0.3333333333333333 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 73, + "SPLIT": 0, + "TEXT_ERROR": 64, + "WELD": 0 + }, + "X": { + "OK": 73, + "SPLIT": 0, + "TEXT_ERROR": 64, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 7, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.14285714285714285 + }, + "X": { + "denominator": 7, + "metric": "M3", + "numerator": 1, + "status": "REPORTED", + "value": 0.14285714285714285 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 6, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 1, + "metric": "M4", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "X": { + "denominator": 1, + "metric": "M4", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_off_region": 2, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 1, + "metric": "M5", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 2 + }, + "X": { + "agreement": { + "denominator": 1, + "metric": "M5", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 2 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 1, + "X": 1 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 3, + "X": 3 + }, + "m3_boundary": { + "H": { + "OK": 73, + "SPLIT": 0, + "TEXT_ERROR": 64, + "WELD": 0 + }, + "X": { + "OK": 73, + "SPLIT": 0, + "TEXT_ERROR": 64, + "WELD": 0 + } + }, + "m3_clean": { + "H": 1, + "X": 1 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 1, + "BOTH_DIRTY": 6, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 7, + "X": 7 + }, + "m4_correct": { + "H": 1, + "X": 1 + }, + "m4_excluded_off_region": 2, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 1, + "X": 1 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 1, + "X": 1 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 2, + "X": 2 + }, + "m5_scored": { + "H": 1, + "X": 1 + }, + "n_adjudicated": 7, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 4, + "X": 4 + }, + "n_matched": { + "H": 3, + "X": 3 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 1 + }, + "116-hr-7617": { + "M0": { + "M0_any": 6922, + "M0_any_rate": { + "denominator": 29173, + "metric": "M0-any", + "numerator": 6922, + "status": "REPORTED", + "value": 0.2372741918897611 + }, + "M0_any_rate_ALL_LINES_superseded": 0.212586, + "M0a_only_text": 6922, + "M0a_text": 6922, + "M0a_text_rate": { + "denominator": 29173, + "metric": "M0a", + "numerator": 6922, + "status": "REPORTED", + "value": 0.2372741918897611 + }, + "M0b_defined": 29173, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 29173, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 29173, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 882, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 4670, + "metric": "M0c", + "numerator": 882, + "status": "REPORTED", + "value": 0.18886509635974305 + }, + "both_absent": 3388, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 32561, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 29173, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 1810, + "rate": { + "denominator": 1810, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 1810, + "rate": { + "denominator": 1810, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 22896, + "n_margin_numbered_lines": 20732, + "n_margin_numbered_with_glyph_size": 20732, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 29170, + "n_margin_numbered_lines": 26946, + "n_margin_numbered_with_glyph_size": 26946, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "116-hr-7617", + "margin_line_loss": { + "deficit": 6214, + "fires": true, + "h": 20732, + "loser": "H", + "x": 26946 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 4, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 0.75 + }, + "recall": { + "denominator": 5, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 0.6 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 5, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 0.6 + } + }, + "X": { + "precision": { + "denominator": 4, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 0.75 + }, + "recall": { + "denominator": 5, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 0.6 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 5, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 0.6 + } + } + }, + "M2": { + "H": { + "denominator": 3, + "metric": "M2", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + }, + "X": { + "denominator": 3, + "metric": "M2", + "numerator": 2, + "status": "REPORTED", + "value": 0.6666666666666666 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 48, + "SPLIT": 0, + "TEXT_ERROR": 18, + "WELD": 0 + }, + "X": { + "OK": 48, + "SPLIT": 0, + "TEXT_ERROR": 18, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 5, + "metric": "M3", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + }, + "X": { + "denominator": 5, + "metric": "M3", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 2, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 4, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 3, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 3, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 2, + "X": 2 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 3, + "X": 3 + }, + "m3_boundary": { + "H": { + "OK": 48, + "SPLIT": 0, + "TEXT_ERROR": 18, + "WELD": 0 + }, + "X": { + "OK": 48, + "SPLIT": 0, + "TEXT_ERROR": 18, + "WELD": 0 + } + }, + "m3_clean": { + "H": 2, + "X": 2 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 2, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 5, + "X": 5 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 4, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 3, + "X": 3 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 3, + "X": 3 + }, + "n_adjudicated": 5, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 4, + "X": 4 + }, + "n_matched": { + "H": 3, + "X": 3 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 7 + }, + "117-hr-3237": { + "M0": { + "M0_any": 145, + "M0_any_rate": { + "denominator": 705, + "metric": "M0-any", + "numerator": 145, + "status": "REPORTED", + "value": 0.20567375886524822 + }, + "M0_any_rate_ALL_LINES_superseded": 0.197817, + "M0a_only_text": 145, + "M0a_text": 145, + "M0a_text_rate": { + "denominator": 705, + "metric": "M0a", + "numerator": 145, + "status": "REPORTED", + "value": 0.20567375886524822 + }, + "M0b_defined": 705, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 705, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 705, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 1, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 109, + "metric": "M0c", + "numerator": 1, + "status": "REPORTED", + "value": 0.009174311926605505 + }, + "both_absent": 28, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 733, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 705, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 79, + "rate": { + "denominator": 79, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 79, + "rate": { + "denominator": 79, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 560, + "n_margin_numbered_lines": 523, + "n_margin_numbered_with_glyph_size": 523, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 704, + "n_margin_numbered_lines": 664, + "n_margin_numbered_with_glyph_size": 664, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "117-hr-3237", + "margin_line_loss": { + "deficit": 141, + "fires": true, + "h": 523, + "loser": "H", + "x": 664 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 3, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 3, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 3, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + } + }, + "X": { + "precision": { + "denominator": 3, + "metric": "M1 precision", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 3, + "metric": "M1 recall", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 3, + "metric": "M1 recall (SECONDARY)", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + } + } + }, + "M2": { + "H": { + "denominator": 3, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 3, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + }, + "X": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 3, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 3, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 3, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 3, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 3, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 0, + "X": 0 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 3, + "X": 3 + }, + "m3_boundary": { + "H": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + }, + "X": { + "OK": 18, + "SPLIT": 0, + "TEXT_ERROR": 3, + "WELD": 0 + } + }, + "m3_clean": { + "H": 0, + "X": 0 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 3, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 3, + "X": 3 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 3, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 3, + "X": 3 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 3, + "X": 3 + }, + "n_adjudicated": 3, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 3, + "X": 3 + }, + "n_matched": { + "H": 3, + "X": 3 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 3 + }, + "117-s-4663": { + "M0": { + "M0_any": 861, + "M0_any_rate": { + "denominator": 3188, + "metric": "M0-any", + "numerator": 861, + "status": "REPORTED", + "value": 0.27007528230865746 + }, + "M0_any_rate_ALL_LINES_superseded": 0.240369, + "M0a_only_text": 861, + "M0a_text": 861, + "M0a_text_rate": { + "denominator": 3188, + "metric": "M0a", + "numerator": 861, + "status": "REPORTED", + "value": 0.27007528230865746 + }, + "M0b_defined": 3188, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 3188, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 3188, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 516, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 394, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 3582, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 3188, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 221, + "rate": { + "denominator": 221, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 221, + "rate": { + "denominator": 221, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 2356, + "n_margin_numbered_lines": 2341, + "n_margin_numbered_with_glyph_size": 2341, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 3188, + "n_margin_numbered_lines": 3173, + "n_margin_numbered_with_glyph_size": 3173, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "117-s-4663", + "margin_line_loss": { + "deficit": 832, + "fires": true, + "h": 2341, + "loser": "H", + "x": 3173 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 5, + "metric": "M1 precision", + "numerator": 4, + "status": "REPORTED", + "value": 0.8 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + } + }, + "X": { + "precision": { + "denominator": 5, + "metric": "M1 precision", + "numerator": 4, + "status": "REPORTED", + "value": 0.8 + }, + "recall": { + "denominator": 7, + "metric": "M1 recall", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 7, + "metric": "M1 recall (SECONDARY)", + "numerator": 4, + "status": "REPORTED", + "value": 0.5714285714285714 + } + } + }, + "M2": { + "H": { + "denominator": 4, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 4, + "metric": "M2", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 59, + "SPLIT": 0, + "TEXT_ERROR": 62, + "WELD": 0 + }, + "X": { + "OK": 59, + "SPLIT": 0, + "TEXT_ERROR": 62, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 7, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "X": { + "denominator": 7, + "metric": "M3", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 7, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "X": { + "denominator": 0, + "metric": "M4", + "numerator": 0, + "status": "VACUOUS", + "value": null + }, + "excluded_off_region": 6, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 4, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 0.75 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 4, + "metric": "M5", + "numerator": 3, + "status": "REPORTED", + "value": 0.75 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 0, + "X": 0 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 4, + "X": 4 + }, + "m3_boundary": { + "H": { + "OK": 59, + "SPLIT": 0, + "TEXT_ERROR": 62, + "WELD": 0 + }, + "X": { + "OK": 59, + "SPLIT": 0, + "TEXT_ERROR": 62, + "WELD": 0 + } + }, + "m3_clean": { + "H": 0, + "X": 0 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 0, + "BOTH_DIRTY": 7, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 7, + "X": 7 + }, + "m4_correct": { + "H": 0, + "X": 0 + }, + "m4_excluded_off_region": 6, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 0, + "X": 0 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 3, + "X": 3 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 4, + "X": 4 + }, + "n_adjudicated": 7, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 5, + "X": 5 + }, + "n_matched": { + "H": 4, + "X": 4 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 7 + }, + "119-hr-6938": { + "M0": { + "M0_any": 897, + "M0_any_rate": { + "denominator": 8871, + "metric": "M0-any", + "numerator": 897, + "status": "REPORTED", + "value": 0.10111599594183295 + }, + "M0_any_rate_ALL_LINES_superseded": 0.101116, + "M0a_only_text": 897, + "M0a_text": 897, + "M0a_text_rate": { + "denominator": 8871, + "metric": "M0a", + "numerator": 897, + "status": "REPORTED", + "value": 0.10111599594183295 + }, + "M0b_defined": 8871, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 8871, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 8871, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 1186, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 0, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 8871, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 8871, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 0.9411764705882353, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 7987, + "n_margin_numbered_lines": 34, + "n_margin_numbered_with_glyph_size": 32, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 0.9411764705882353, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 8869, + "n_margin_numbered_lines": 34, + "n_margin_numbered_with_glyph_size": 32, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": false, + "loser": null, + "x": false + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "119-hr-6938", + "margin_line_loss": { + "deficit": 0, + "fires": false, + "h": 34, + "loser": null, + "x": 34 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 4 + }, + "119-hr-7148": { + "M0": { + "M0_any": 3310, + "M0_any_rate": { + "denominator": 30548, + "metric": "M0-any", + "numerator": 3310, + "status": "REPORTED", + "value": 0.10835406573261752 + }, + "M0_any_rate_ALL_LINES_superseded": 0.108354, + "M0a_only_text": 3310, + "M0a_text": 3310, + "M0a_text_rate": { + "denominator": 30548, + "metric": "M0a", + "numerator": 3310, + "status": "REPORTED", + "value": 0.10835406573261752 + }, + "M0b_defined": 30548, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 30548, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 30548, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 4101, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 0, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 30548, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 30548, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 0.9259259259259259, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 27293, + "n_margin_numbered_lines": 81, + "n_margin_numbered_with_glyph_size": 75, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 0.9259259259259259, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 30534, + "n_margin_numbered_lines": 81, + "n_margin_numbered_with_glyph_size": 75, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": false, + "loser": null, + "x": false + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "119-hr-7148", + "margin_line_loss": { + "deficit": 0, + "fires": false, + "h": 81, + "loser": null, + "x": 81 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 4 + }, + "119-hr-8469": { + "M0": { + "M0_any": 592, + "M0_any_rate": { + "denominator": 2281, + "metric": "M0-any", + "numerator": 592, + "status": "REPORTED", + "value": 0.2595352915387988 + }, + "M0_any_rate_ALL_LINES_superseded": 0.230979, + "M0a_only_text": 592, + "M0a_text": 592, + "M0a_text_rate": { + "denominator": 2281, + "metric": "M0a", + "numerator": 592, + "status": "REPORTED", + "value": 0.2595352915387988 + }, + "M0b_defined": 2281, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 2281, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 2281, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 373, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 282, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 2563, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 2281, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 195, + "rate": { + "denominator": 195, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 195, + "rate": { + "denominator": 195, + "metric": "M7", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 1714, + "n_margin_numbered_lines": 1695, + "n_margin_numbered_with_glyph_size": 1695, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": true, + "n_lines_total": 2281, + "n_margin_numbered_lines": 2261, + "n_margin_numbered_with_glyph_size": 2261, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "119-hr-8469", + "margin_line_loss": { + "deficit": 566, + "fires": true, + "h": 1695, + "loser": "H", + "x": 2261 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "headings_by_frame": { + "C": { + "M1": { + "H": { + "precision": { + "denominator": 5, + "metric": "M1 precision", + "numerator": 5, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 6, + "metric": "M1 recall", + "numerator": 5, + "status": "REPORTED", + "value": 0.8333333333333334 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 6, + "metric": "M1 recall (SECONDARY)", + "numerator": 5, + "status": "REPORTED", + "value": 0.8333333333333334 + } + }, + "X": { + "precision": { + "denominator": 5, + "metric": "M1 precision", + "numerator": 5, + "status": "REPORTED", + "value": 1.0 + }, + "recall": { + "denominator": 6, + "metric": "M1 recall", + "numerator": 5, + "status": "REPORTED", + "value": 0.8333333333333334 + }, + "recall_sensitivity_excluding_unresolvable": { + "denominator": 6, + "metric": "M1 recall (SECONDARY)", + "numerator": 5, + "status": "REPORTED", + "value": 0.8333333333333334 + } + } + }, + "M2": { + "H": { + "denominator": 5, + "metric": "M2", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + }, + "X": { + "denominator": 5, + "metric": "M2", + "numerator": 2, + "status": "REPORTED", + "value": 0.4 + } + }, + "M3": { + "boundary_outcomes": { + "H": { + "OK": 78, + "SPLIT": 0, + "TEXT_ERROR": 12, + "WELD": 0 + }, + "X": { + "OK": 78, + "SPLIT": 0, + "TEXT_ERROR": 12, + "WELD": 0 + } + }, + "clean_rate": { + "H": { + "denominator": 6, + "metric": "M3", + "numerator": 2, + "status": "REPORTED", + "value": 0.3333333333333333 + }, + "X": { + "denominator": 6, + "metric": "M3", + "numerator": 2, + "status": "REPORTED", + "value": 0.3333333333333333 + } + }, + "consumes": "projected text + oracle only -- never a segmentation label (section 5)", + "decision_unit": "heading occurrence (A3); WELD/SPLIT tallies never substitute for it", + "heading_outcomes": { + "BOTH_CLEAN": 2, + "BOTH_DIRTY": 4, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + } + }, + "M4": { + "H": { + "denominator": 1, + "metric": "M4", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "X": { + "denominator": 1, + "metric": "M4", + "numerator": 1, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_off_region": 4, + "excluded_unreadable": 0, + "h_correct_x_wrong": 0, + "h_correct_x_wrong_occurrences": [], + "paired_population": "headings scored under M4 for BOTH arms; an unmatched heading is M1 recall's miss and M3's TEXT_ERROR, never charged twice here", + "parent_answers": "printed text | NONE | OFF_REGION | UNREADABLE", + "scope": "immediate parent only (A38.4 penultimate breadcrumb element)", + "veto_owner": "decide_architecture -- Rule 1 condition 4; no consequence is applied here", + "x_correct_h_wrong": 0 + }, + "M5": { + "H": { + "agreement": { + "denominator": 5, + "metric": "M5", + "numerator": 5, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "X": { + "agreement": { + "denominator": 5, + "metric": "M5", + "numerator": 5, + "status": "REPORTED", + "value": 1.0 + }, + "excluded_unscorable": 0 + }, + "licenses": "corroboration only -- section 6 forbids M5 deciding anything", + "r1_role_gate": { + "evidence": "computed from the committed oracle key + adjudications, never supplied", + "m5_void": false, + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "PASS", + "threshold": 0.8 + } + }, + "counts": { + "m2_exact": { + "H": 2, + "X": 2 + }, + "m2_excluded_unreadable": 0, + "m2_scored": { + "H": 5, + "X": 5 + }, + "m3_boundary": { + "H": { + "OK": 78, + "SPLIT": 0, + "TEXT_ERROR": 12, + "WELD": 0 + }, + "X": { + "OK": 78, + "SPLIT": 0, + "TEXT_ERROR": 12, + "WELD": 0 + } + }, + "m3_clean": { + "H": 2, + "X": 2 + }, + "m3_excluded_no_reference": 0, + "m3_outcomes": { + "BOTH_CLEAN": 2, + "BOTH_DIRTY": 4, + "UNSCORABLE": 0, + "X_CORRECTS": 0, + "X_REGRESSES": 0 + }, + "m3_scored": { + "H": 6, + "X": 6 + }, + "m4_correct": { + "H": 1, + "X": 1 + }, + "m4_excluded_off_region": 4, + "m4_excluded_unreadable": 0, + "m4_h_correct_x_wrong": 0, + "m4_h_correct_x_wrong_keys": [], + "m4_scored": { + "H": 1, + "X": 1 + }, + "m4_x_correct_h_wrong": 0, + "m5_agree": { + "H": 5, + "X": 5 + }, + "m5_excluded_unreadable": 0, + "m5_excluded_unscorable": { + "H": 0, + "X": 0 + }, + "m5_scored": { + "H": 5, + "X": 5 + }, + "n_adjudicated": 6, + "n_adjudicated_unresolvable": 0, + "n_emitted": { + "H": 5, + "X": 5 + }, + "n_matched": { + "H": 5, + "X": 5 + }, + "n_stimuli": 8 + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "r1_text_gate": { + "observed_by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "owner": "Rule 3 gate vector (A27.6); this module reports the FACTS only", + "status": "FAIL", + "threshold": 0.9, + "voids_text_metrics_if_FAIL": [ + "M2", + "M3" + ] + } + } + }, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 7 + }, + "CRPT-114HRPT215": { + "M0": { + "M0_any": 1913, + "M0_any_rate": { + "denominator": 8120, + "metric": "M0-any", + "numerator": 1913, + "status": "REPORTED", + "value": 0.2355911330049261 + }, + "M0_any_rate_ALL_LINES_superseded": 0.224425, + "M0a_only_text": 1913, + "M0a_text": 1913, + "M0a_text_rate": { + "denominator": 8120, + "metric": "M0a", + "numerator": 1913, + "status": "REPORTED", + "value": 0.2355911330049261 + }, + "M0b_defined": 8120, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 8120, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 8120, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 1168, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 404, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 8524, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 8120, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 6239, + "n_margin_numbered_lines": 11, + "n_margin_numbered_with_glyph_size": 11, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 8111, + "n_margin_numbered_lines": 11, + "n_margin_numbered_with_glyph_size": 11, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": false, + "loser": null, + "x": false + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "CRPT-114HRPT215", + "margin_line_loss": { + "deficit": 0, + "fires": false, + "h": 11, + "loser": null, + "x": 11 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 6 + }, + "CRPT-114HRPT605": { + "M0": { + "M0_any": 1308, + "M0_any_rate": { + "denominator": 5284, + "metric": "M0-any", + "numerator": 1308, + "status": "REPORTED", + "value": 0.24753974261922787 + }, + "M0_any_rate_ALL_LINES_superseded": 0.236357, + "M0a_only_text": 1308, + "M0a_text": 1308, + "M0a_text_rate": { + "denominator": 5284, + "metric": "M0a", + "numerator": 1308, + "status": "REPORTED", + "value": 0.24753974261922787 + }, + "M0b_defined": 5284, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 5284, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 5284, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 747, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 250, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 5534, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 5284, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 4012, + "n_margin_numbered_lines": 8, + "n_margin_numbered_with_glyph_size": 8, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 5273, + "n_margin_numbered_lines": 8, + "n_margin_numbered_with_glyph_size": 8, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": false, + "loser": null, + "x": false + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "CRPT-114HRPT605", + "margin_line_loss": { + "deficit": 0, + "fires": false, + "h": 8, + "loser": null, + "x": 8 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 6 + }, + "CRPT-119HRPT632": { + "M0": { + "M0_any": 2353, + "M0_any_rate": { + "denominator": 9496, + "metric": "M0-any", + "numerator": 2353, + "status": "REPORTED", + "value": 0.24778854254422916 + }, + "M0_any_rate_ALL_LINES_superseded": 0.234596, + "M0a_only_text": 2353, + "M0a_text": 2353, + "M0a_text_rate": { + "denominator": 9496, + "metric": "M0a", + "numerator": 2353, + "status": "REPORTED", + "value": 0.24778854254422916 + }, + "M0b_defined": 9496, + "M0b_only_segmentation": 0, + "M0b_rate_on_defined": { + "denominator": 9496, + "metric": "M0b_on_defined", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0b_segmentation": 0, + "M0b_segmentation_rate": { + "denominator": 9496, + "metric": "M0b_on_risk_set", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "M0c_anchor_regions": 0, + "M0c_pooling": "REGION-level; never pooled with the line rates (section 5)", + "M0c_rate": { + "denominator": 1415, + "metric": "M0c", + "numerator": 0, + "status": "REPORTED", + "value": 0.0 + }, + "both_absent": 534, + "both_absent_in_any_denominator": false, + "neutral_lines_in_scope": 10030, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "risk_set": 9496, + "risk_set_definition": "neutral lines with state != BOTH_ABSENT (A23, I3)" + }, + "M7": { + "H": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "X": { + "instances": [], + "n_display_split": 0, + "n_emitted": 0, + "rate": { + "denominator": 0, + "metric": "M7", + "numerator": 0, + "status": "VACUOUS", + "value": null + } + }, + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "threshold_single_char_tokens": 3 + }, + "M9": { + "H": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 7174, + "n_margin_numbered_lines": 9, + "n_margin_numbered_with_glyph_size": 9, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "X": { + "coverage": 1.0, + "coverage_floor": 0.85, + "coverage_meets_floor": true, + "derive_size_bands_returns_a_band": false, + "n_lines_total": 9487, + "n_margin_numbered_lines": 9, + "n_margin_numbered_with_glyph_size": 9, + "rule0_comparison": "RAW FACTS ONLY -- Rule 0 margin-line interpretation frozen by A39.1 (count of Page.lines where line_number is not None; any positive deficit fires); application belongs to the later architecture decision" + }, + "band_loss": { + "fires": false, + "h": false, + "loser": null, + "x": false + }, + "coverage_floor": 0.85, + "coverage_loss": { + "fires": false, + "h": true, + "loser": null, + "x": true + }, + "document": "CRPT-119HRPT632", + "margin_line_loss": { + "deficit": 0, + "fires": false, + "h": 9, + "loser": null, + "x": 9 + }, + "margin_line_quantity": "count of Page.lines where line_number is not None (A39.1); no tolerance", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "rule0_outcome": null, + "rule0_owner": "decide_architecture -- A27.4's outcome enum is deliberately absent here" + }, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)", + "stratum": 6 + } + }, + "r1_reliability": { + "a48_required_routes_changed": true, + "aggregation": "heading-occurrence micro-average per route; worst required route (R6.4)", + "confirmatory_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A48 post-boundary deviation)", + "decision_owner": "Rule 3 gate vector (A27.6); no consequence is applied here", + "matching": "one-to-one on uniquely resolved A30 occurrence keys; symmetric union denominator (R6.1)", + "n_pairs": 6, + "pairs": [ + { + "denominator": 1, + "n_matched_pairs": 1, + "n_primary_headings": 1, + "n_primary_non_unique": 0, + "n_primary_unresolved": 0, + "n_repeat_headings": 1, + "n_repeat_non_unique": 0, + "n_repeat_unresolved": 0, + "n_role_agree": 1, + "n_text_agree": 1, + "primary_blind_id": "22d6fd3de1ba7837", + "repeat_blind_id": "2e7e0d21d4ab1769", + "route": "ai" + }, + { + "denominator": 1, + "n_matched_pairs": 1, + "n_primary_headings": 1, + "n_primary_non_unique": 0, + "n_primary_unresolved": 0, + "n_repeat_headings": 1, + "n_repeat_non_unique": 0, + "n_repeat_unresolved": 0, + "n_role_agree": 1, + "n_text_agree": 1, + "primary_blind_id": "9bf5371acb85cb21", + "repeat_blind_id": "5b3e82d843a78143", + "route": "ai" + }, + { + "denominator": 1, + "n_matched_pairs": 0, + "n_primary_headings": 0, + "n_primary_non_unique": 0, + "n_primary_unresolved": 0, + "n_repeat_headings": 1, + "n_repeat_non_unique": 0, + "n_repeat_unresolved": 0, + "n_role_agree": 0, + "n_text_agree": 0, + "primary_blind_id": "265fdcf236211f36", + "repeat_blind_id": "8f6286a669bc6f38", + "route": "ai" + }, + { + "denominator": 1, + "n_matched_pairs": 1, + "n_primary_headings": 1, + "n_primary_non_unique": 0, + "n_primary_unresolved": 0, + "n_repeat_headings": 1, + "n_repeat_non_unique": 0, + "n_repeat_unresolved": 0, + "n_role_agree": 1, + "n_text_agree": 1, + "primary_blind_id": "cda24e7a522b27f7", + "repeat_blind_id": "e500d7a18f808831", + "route": "ai" + }, + { + "denominator": 1, + "n_matched_pairs": 1, + "n_primary_headings": 1, + "n_primary_non_unique": 0, + "n_primary_unresolved": 0, + "n_repeat_headings": 1, + "n_repeat_non_unique": 0, + "n_repeat_unresolved": 0, + "n_role_agree": 1, + "n_text_agree": 1, + "primary_blind_id": "e17cc1a6f7ee5413", + "repeat_blind_id": "f05a2796e9ccb24e", + "route": "ai" + }, + { + "denominator": 0, + "n_matched_pairs": 0, + "n_primary_headings": 0, + "n_primary_non_unique": 0, + "n_primary_unresolved": 0, + "n_repeat_headings": 0, + "n_repeat_non_unique": 0, + "n_repeat_unresolved": 0, + "n_role_agree": 0, + "n_text_agree": 0, + "primary_blind_id": "77e97a265604c03f", + "repeat_blind_id": "fc78fac0ea40f61b", + "route": "ai" + } + ], + "per_route": { + "ai": { + "n_pairs": 6, + "role": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + }, + "text": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + } + }, + "pooled_across_routes": false, + "role": { + "by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "PASS", + "threshold": 0.8 + } + }, + "status": "PASS", + "threshold": 0.8 + }, + "role_comparison": "exact fine section 5.3 role; UNREADABLE never agrees (R6.3)", + "ruled_by": "A41.2 R6", + "text": { + "by_route": { + "ai": { + "denominator": 5, + "numerator": 4, + "ratio": 0.8, + "status": "FAIL", + "threshold": 0.9 + } + }, + "status": "FAIL", + "threshold": 0.9 + }, + "text_comparison": "EXACT equality of the values as returned -- no normalisation (R6.2)" + }, + "s1": { + "advance_scale": 1.25, + "fires": true, + "gate_owner": "Rule 3 gate vector (A27.6)", + "m0_reportable": true, + "m0_status": "REPORTED" + }, + "schema": "metrics/1", + "section8": { + "bootstrap": { + "endpoint_indices": [ + 249, + 9749 + ], + "events": 5, + "gating": false, + "interval": [ + 0.16666666666666666, + 0.6666666666666666 + ], + "n_documents": 12, + "namespace": "bootstrap-document", + "observed_rate": 0.4166666666666667, + "reported": true, + "resamples": 10000, + "seed": 20260807, + "statistic": [ + "section8", + "document-heading-discordance", + "P-head" + ], + "unit": "document" + }, + "clopper_pearson_alpha": 0.05, + "clopper_pearson_upper_bound": 0.6847622092755479, + "estimand": "pi = P(a document from the target population shows >= 1 heading-level H/X discordance)", + "events": 5, + "excluded_diagnostics_not_in_statistic": [ + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "119-hr-6938", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "119-hr-7148", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "CRPT-114HRPT215", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "CRPT-114HRPT605", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "CRPT-119HRPT632", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + } + ], + "excluded_documents_not_p_head": [ + "119-hr-6938", + "119-hr-7148", + "CRPT-114HRPT215", + "CRPT-114HRPT605", + "CRPT-119HRPT632" + ], + "forbidden": "no per-heading probability; no heading-as-iid-trial denominator (section 8.3)", + "independent_unit": "document", + "n_documents": 12, + "paired": { + "by_population": { + "P-head": { + "coverage": { + "n_documents": 12, + "per_document": [ + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "113-hr-933", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "114-s-3001", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "115-hr-5961", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "115-hr-6147", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "115-hr-6157", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "115-s-1609", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "115-s-2976", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "116-hr-7611", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "116-hr-7617", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "117-hr-3237", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "117-s-4663", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "119-hr-8469", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + } + ], + "status": "REPORTED", + "unweighted_mean_difference": 0.0 + }, + "n_margin_numbered_lines": { + "n_documents": 12, + "per_document": [ + { + "H": 10825, + "X": 14098, + "difference": 3273, + "document": "113-hr-933", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1420, + "X": 1826, + "difference": 406, + "document": "114-s-3001", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 2197, + "X": 2886, + "difference": 689, + "document": "115-hr-5961", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 2760, + "X": 3570, + "difference": 810, + "document": "115-hr-6147", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 6403, + "X": 8243, + "difference": 1840, + "document": "115-hr-6157", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1162, + "X": 1535, + "difference": 373, + "document": "115-s-1609", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1872, + "X": 2431, + "difference": 559, + "document": "115-s-2976", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 835, + "X": 1049, + "difference": 214, + "document": "116-hr-7611", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 20732, + "X": 26946, + "difference": 6214, + "document": "116-hr-7617", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 523, + "X": 664, + "difference": 141, + "document": "117-hr-3237", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 2341, + "X": 3173, + "difference": 832, + "document": "117-s-4663", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1695, + "X": 2261, + "difference": 566, + "document": "119-hr-8469", + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + } + ], + "status": "REPORTED", + "unweighted_mean_difference": 1326.4166666666667 + } + }, + "P-robust": { + "coverage": { + "n_documents": 5, + "per_document": [ + { + "H": 0.9411764705882353, + "X": 0.9411764705882353, + "difference": 0.0, + "document": "119-hr-6938", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 0.9259259259259259, + "X": 0.9259259259259259, + "difference": 0.0, + "document": "119-hr-7148", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "CRPT-114HRPT215", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "CRPT-114HRPT605", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 1.0, + "X": 1.0, + "difference": 0.0, + "document": "CRPT-119HRPT632", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + } + ], + "status": "REPORTED", + "unweighted_mean_difference": 0.0 + }, + "n_margin_numbered_lines": { + "n_documents": 5, + "per_document": [ + { + "H": 34, + "X": 34, + "difference": 0, + "document": "119-hr-6938", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 81, + "X": 81, + "difference": 0, + "document": "119-hr-7148", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 11, + "X": 11, + "difference": 0, + "document": "CRPT-114HRPT215", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 8, + "X": 8, + "difference": 0, + "document": "CRPT-114HRPT605", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "H": 9, + "X": 9, + "difference": 0, + "document": "CRPT-119HRPT632", + "population": "P-robust", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + } + ], + "status": "REPORTED", + "unweighted_mean_difference": 0.0 + } + } + }, + "difference": "X - H", + "excluded_quantities": { + "M1-M5, M7": "can be VACUOUS -- pairing them would require an unruled missingness policy", + "coverage_floor": "a frozen constant, so its difference is always zero", + "derive_size_bands_returns_a_band, coverage_meets_floor": "booleans consumed by Rule 0, not numeric", + "n_margin_numbered_with_glyph_size": "a diagnostic; A39.1's quantity is the margin-line count" + }, + "no_combined_mean": "P-head and P-robust are reported separately; no 17-document mean exists", + "populations_never_pooled": true, + "quantities": [ + "n_margin_numbered_lines", + "coverage" + ], + "quantity_source": "frames.m9.{H,X} -- the numeric M9 bases (A38.8/A39.1)", + "ruled_by": "A41.2 R9", + "weighting": "UNWEIGHTED mean over documents (\u00a78.3); never weighted by heading count" + }, + "per_document": [ + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "113-hr-933", + "event": true, + "evidence": [ + { + "page_number": 4, + "region_ordinal": 1 + }, + { + "page_number": 6, + "region_ordinal": 1 + }, + { + "page_number": 6, + "region_ordinal": 2 + }, + { + "page_number": 7, + "region_ordinal": 0 + }, + { + "page_number": 7, + "region_ordinal": 1 + }, + { + "page_number": 7, + "region_ordinal": 2 + }, + { + "page_number": 9, + "region_ordinal": 0 + }, + { + "page_number": 9, + "region_ordinal": 1 + } + ], + "n_anchor_discordant_regions": 367, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "114-s-3001", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "115-hr-5961", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "115-hr-6147", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "115-hr-6157", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "115-s-1609", + "event": true, + "evidence": [ + { + "page_number": 38, + "region_ordinal": 1 + } + ], + "n_anchor_discordant_regions": 1, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "115-s-2976", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "116-hr-7611", + "event": true, + "evidence": [ + { + "page_number": 8, + "region_ordinal": 1 + } + ], + "n_anchor_discordant_regions": 1, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "116-hr-7617", + "event": true, + "evidence": [ + { + "page_number": 319, + "region_ordinal": 1 + }, + { + "page_number": 319, + "region_ordinal": 2 + }, + { + "page_number": 320, + "region_ordinal": 1 + }, + { + "page_number": 321, + "region_ordinal": 1 + }, + { + "page_number": 321, + "region_ordinal": 2 + }, + { + "page_number": 323, + "region_ordinal": 1 + }, + { + "page_number": 323, + "region_ordinal": 2 + }, + { + "page_number": 324, + "region_ordinal": 0 + } + ], + "n_anchor_discordant_regions": 882, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "117-hr-3237", + "event": true, + "evidence": [ + { + "page_number": 24, + "region_ordinal": 0 + } + ], + "n_anchor_discordant_regions": 1, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "117-s-4663", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + }, + { + "definition": "any region whose emitted heading (Anchor) sets differ between H and X", + "document": "119-hr-8469", + "event": false, + "evidence": [], + "n_anchor_discordant_regions": 0, + "population": "P-head", + "qualification": null, + "qualification_status": "NON-CONFIRMATORY (PRE-REGISTRATION 4.7 -- A45 post-boundary deviation)" + } + ], + "population": "P-head", + "statement": "5 of 12 P-head document(s) showed at least one heading-level discordance; exact one-sided 95 % upper bound on the per-document rate: 0.6848.", + "status": "REPORTED", + "zero_event_closed_form_used": false + } +} \ No newline at end of file