Reconcile the verdict with the measurement (#32) - #48
Merged
Conversation
Part two of #32. The wording, now that the boundary is settled. Four bands become two, because two is what the corpus can support. "Strong", "Moderate" and "Light" read as three measured degrees and never were. The calibration locates one boundary and says nothing whatever about 45 or 70: no text known to be human came within twenty points of either, so the corpus cannot place them, and placing them would need machine-written text — which Docs/CALIBRATION.md argues at length against ever collecting, since any such collection samples whichever models were convenient that month and flatters whoever assembled it. Issue #32 offered keeping the bands with a footnote admitting they are unmeasured. That is worse than dropping them: the footnote is read once and the heading is read every time. Above the boundary the page now says "Signs of AI writing" and lets the findings carry the weight, which is what the findings are for. Below it, the report said "Reads mostly human" and the interface said "Minimal signs of AI writing" — the same state, two different claims, and only one of them ours to make. Both now read "No signs above the measured boundary". A detector that detects nothing also returns a low score, and this project has deliberately never measured how much machine writing it catches, so a low score is evidence about where our boundary sits and about nothing else. It was the last place the product still made a claim about a person rather than about itself. The colour cuts at 45 and 70 survive as exactly that: a display convention, documented as unmeasured where a reader of the code will meet it, making no claim in any language. Rewording the English retires the SHA-256 pin each translation records for it, so the Spanish moves in the same commit — otherwise every pin goes stale and the whole report silently falls back to English, undoing #36 by accident. There is now a test that renders the Spanish report and fails if that happens. Two existing assertions were rewritten rather than deleted: both named strings this commit removes, so both would have passed forever without testing anything.
Part three of #32, and the part that makes the other two checkable by a stranger. The page published a threshold and stopped there, leaving the reader to discover by using the tool that the product drew its line five points lower — which is the same gap this project exists to point out in detectors that publish an accuracy figure and ship something else. A page that prints the figure without saying what is done with it invites exactly that charge. The new section states the boundary, that it is taken from the table above it rather than from anybody's judgement, and that it moves when the page moves — including upward, if a larger corpus turns out to be less flattering than this one. It says there is one verdict above the line and not a scale of them, and why: this corpus can place a boundary and cannot say how far past it a score has travelled, because no text known to be human came near the upper reaches and grading them apart would need machine-written text that the opening of the same page argues against collecting. It names the colour shades as a display convention, on the page, where a reader who wondered why they are absent from every table can find the answer. It is generated rather than written, so a fork that has measured nothing gets the honest version of the same paragraph instead of ours, and a corpus that grows rewrites it without anybody remembering to. Re-ran the calibration before and after, as #32 asks. published-calibration.json is byte-identical: same fingerprint 123fa5b9ebca3f29, same 90 texts, same threshold of 25, same interval of 0% – 4.1%. Bands never fed the score, and this is the evidence rather than the argument — the discipline exists so that nobody can move a boundary and present the improved number as an achievement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #32.
What was wrong
#30published a measured threshold of 25/100. The product never used it.Worse, the three surfaces did not agree with each other. On one text, one engine, one run:
90/100 — Strong signs of AI writingDocs/CALIBRATION.mdEvidenceReport.VerdictHoldsdemanded the per-language threshold and refused to borrow theaggregate. No language in the corpus has one — English has 65 texts and Spanish 25, against the ~75
the interval needs — so the condition was false for every document in every language, and the
exported report had never once carried a verdict. Meanwhile the CLI and the web page printed a
hand-picked band unconditionally, from 20: five points below the published threshold and below the
highest-scoring human text in the corpus (23.4).
The bands were written out in nine places, kept in step by a comment reading "mirrors the bands
in
AnalysisResult.Verdict". They had already drifted —Batch.razorcut at 40 where every othersurface cut at 45 — so one build could colour the same document two ways.
Three commits
1 — Let one place decide when the tool speaks.
VerdictBandsreads the boundary from theembedded calibration; the nine sites go through it.
The distinction that unblocks this: borrowing the aggregate error rate would misstate how often
this build is wrong about Spanish (13.3% measured, 4.1% pooled) and stays forbidden — the caveat
still quotes the language's own bound. Borrowing the aggregate boundary asserts nothing about
reliability; it decides when the tool speaks, it is measured and published, and it is printed beside
the language's own figure.
Three states, because two would have regressed a test that was right:
no bound on the page to correct the impression a verdict leaves
2 — Say only what was measured, and say it about the tool. Four bands become two.
"Strong", "Moderate" and "Light" read as three measured degrees and never were. #32 offered keeping
them with a footnote admitting as much; that is worse than dropping them, because the footnote is
read once and the heading is read every time.
Below the line, the report said "Reads mostly human" and the interface "Minimal signs of AI
writing" — the same state, two claims, and only one of them ours to make. Both now read "No signs
above the measured boundary". It was the last place the product spoke about a person rather than
about itself.
45 and 70 survive as a colour convention, documented as unmeasured where a reader of the code meets
it.
3 — Say it on the calibration page too, generated rather than written, so a fork that measured
nothing gets the honest version of the paragraph instead of ours.
The discipline #32 asked for
Calibration re-run before and after.
published-calibration.jsonis byte-identical: samefingerprint
123fa5b9ebca3f29, same 90 texts, same threshold 25, same interval 0% – 4.1%. Bandsnever fed the score — evidence rather than argument, since the whole point of the rule is that
nobody can move a boundary and present the improved number as an achievement.
Tests
344 pass (265 + 72 + 7). The new
VerdictAgreementTestsfails on the parent commit — it is nota test written to pass.
It also pins a trap worth naming: rewording the English retires the SHA-256 pin each translation
records for it, and a stale pin makes the whole report fall back to English silently, undoing #36
by accident. One test renders the Spanish report and fails if that happens.
Two existing assertions in
EvidenceReportTestswere rewritten rather than deleted — both namedstrings this PR removes, so both would have passed forever without testing anything.
Reviewed before it was built
Designed with the committee at the design stage, adversarially, before any code existed. Both
reviewers independently confirmed the never-a-verdict finding. One proposed falling back to the
aggregate unconditionally, which would have regressed the unmeasured-language case; hence three
states rather than two.
Not done here
The published articles quote the old output (
0/100 Reads mostly human, "35/100. Light signs.").Left alone deliberately: a dated article is a record, and editing it to match today's code
falsifies it.