#28 ships the per-writer comparison with one deliberate hole in it, and this is the hole.
What was left undone
WordsOutsideOwnRange counts how many function words the questioned text uses at a rate the writer
never uses in any of their own pieces. It separates cleanly on the handful of documents measured:
|
words outside the writer's range |
| this repo's own articles, held out one at a time |
0, 1, 1, 3 of ~80 |
| a plainly different voice (18th-century legal English) |
14 of 93 |
A threshold on that share was written, and deleted, because "a quarter" came from looking at five
documents and picking a number that separated them. That is not calibration, and a constant near the
top of a file would have done quiet damage in every case nobody tested.
So the count is reported and a human reads it. Placement stays on the aggregate, on one rule anyone
can restate out loud. That_count_does_not_decide_the_placement keeps it that way.
What would close it honestly
A corpus of texts with known authorship, in both English and Spanish, ideally student-length and
including second-language writers — then measure the distribution of the count for same-author pairs
and different-author pairs and see whether a defensible cut point exists at all.
This is the same corpus work the accuracy question needs (see the conformal-calibration approach in
the research notes): the honest answer to "what is your accuracy" is not a number but a bounded
false-positive rate at a chosen threshold, calibrated locally — and calibrated separately for
Spanish, which is precisely what nobody publishes.
If the distributions overlap, the right outcome is to say so and leave the count where it is.
Smaller follow-ups from #28
AtTheEdge may be doing too little work. With the aggregate diluted, a radically different voice
landed there rather than beyond. Cosine Delta (Evert et al., 2017) reportedly discriminates better
than classic Delta; worth measuring before adopting, since it costs the "distance in the writer's own
units" reading that makes the report legible.
- Chunking is fixed at 350 words. Sensitivity to that has not been measured.
- The minimums (≈1,400 / 300 words) are judgement, not measurement. They err toward refusing to
answer, which is the right direction, but the actual noise floor is unknown.
- Only English and Spanish have function-word lists. They are JSON in the rule packs — adding a
language is a pull request, and a speaker of that language is far better placed to do it than I am.
#28 ships the per-writer comparison with one deliberate hole in it, and this is the hole.
What was left undone
WordsOutsideOwnRangecounts how many function words the questioned text uses at a rate the writernever uses in any of their own pieces. It separates cleanly on the handful of documents measured:
A threshold on that share was written, and deleted, because "a quarter" came from looking at five
documents and picking a number that separated them. That is not calibration, and a constant near the
top of a file would have done quiet damage in every case nobody tested.
So the count is reported and a human reads it. Placement stays on the aggregate, on one rule anyone
can restate out loud.
That_count_does_not_decide_the_placementkeeps it that way.What would close it honestly
A corpus of texts with known authorship, in both English and Spanish, ideally student-length and
including second-language writers — then measure the distribution of the count for same-author pairs
and different-author pairs and see whether a defensible cut point exists at all.
This is the same corpus work the accuracy question needs (see the conformal-calibration approach in
the research notes): the honest answer to "what is your accuracy" is not a number but a bounded
false-positive rate at a chosen threshold, calibrated locally — and calibrated separately for
Spanish, which is precisely what nobody publishes.
If the distributions overlap, the right outcome is to say so and leave the count where it is.
Smaller follow-ups from #28
AtTheEdgemay be doing too little work. With the aggregate diluted, a radically different voicelanded there rather than beyond. Cosine Delta (Evert et al., 2017) reportedly discriminates better
than classic Delta; worth measuring before adopting, since it costs the "distance in the writer's own
units" reading that makes the report legible.
answer, which is the right direction, but the actual noise floor is unknown.
language is a pull request, and a speaker of that language is far better placed to do it than I am.