Skip to content

Baseline: calibrate the outside-the-range count against known authorship #29

Description

@peopleworks

#28 ships the per-writer comparison with one deliberate hole in it, and this is the hole.

What was left undone

WordsOutsideOwnRange counts how many function words the questioned text uses at a rate the writer
never uses in any of their own pieces. It separates cleanly on the handful of documents measured:

words outside the writer's range
this repo's own articles, held out one at a time 0, 1, 1, 3 of ~80
a plainly different voice (18th-century legal English) 14 of 93

A threshold on that share was written, and deleted, because "a quarter" came from looking at five
documents and picking a number that separated them. That is not calibration, and a constant near the
top of a file would have done quiet damage in every case nobody tested.

So the count is reported and a human reads it. Placement stays on the aggregate, on one rule anyone
can restate out loud. That_count_does_not_decide_the_placement keeps it that way.

What would close it honestly

A corpus of texts with known authorship, in both English and Spanish, ideally student-length and
including second-language writers — then measure the distribution of the count for same-author pairs
and different-author pairs and see whether a defensible cut point exists at all.

This is the same corpus work the accuracy question needs (see the conformal-calibration approach in
the research notes): the honest answer to "what is your accuracy" is not a number but a bounded
false-positive rate at a chosen threshold, calibrated locally
— and calibrated separately for
Spanish, which is precisely what nobody publishes.

If the distributions overlap, the right outcome is to say so and leave the count where it is.

Smaller follow-ups from #28

  • AtTheEdge may be doing too little work. With the aggregate diluted, a radically different voice
    landed there rather than beyond. Cosine Delta (Evert et al., 2017) reportedly discriminates better
    than classic Delta; worth measuring before adopting, since it costs the "distance in the writer's own
    units" reading that makes the report legible.
  • Chunking is fixed at 350 words. Sensitivity to that has not been measured.
  • The minimums (≈1,400 / 300 words) are judgement, not measurement. They err toward refusing to
    answer, which is the right direction, but the actual noise floor is unknown.
  • Only English and Spanish have function-word lists. They are JSON in the rule packs — adding a
    language is a pull request, and a speaker of that language is far better placed to do it than I am.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions