Tracking the loose ends left by #24, so they are written down somewhere other than a code comment.
1. Cherokee confusables are missing
The lookalike table covers Cyrillic, Greek, Armenian, mathematical alphanumerics and fullwidth forms.
Cherokee is used by substitution tools and was left out on purpose rather than guessed at — a
wrong mapping in this table is worse than a missing one, because it produces a false statement about
someone's file. Adding it needs the codepoints checked against the standard, not recalled.
Same for any other script someone can verify: the table is one dictionary in
src/SignsOfAI.Core/Artifacts/Characters.cs.
2. PDF text extraction may not see what the reader sees
There is published work on attacking detectors through the PDF layer alone
(arXiv 2508.01887): the extracted text can differ from the
rendered text without either being modified. This touches SignsOfAI.Documents, which reads PDF
on the desktop.
Worth checking what PdfPig does with invisible text layers, off-page text, and fonts with remapped
encodings — and whether the scanner should say anything when extraction and rendering disagree. Not
investigated yet.
3. Non-ASCII Latin lookalikes are only partly covered
ı, ɑ, ɡ, ɩ are in the table. Small-capital and other Latin Extended forms are not. These are
harder than the cross-script cases because they are Latin, so the mixed-script signal does not
help — the majority-ASCII rule inside a letter run is doing all the work.
4. A false positive is the report I most want
The tool has no server and no telemetry, so a human telling us is the only way we ever find out. If
a document gets flagged and shouldn't be, please open it under False positives in Discussions
with the codepoint and enough context to reproduce.
Tracking the loose ends left by #24, so they are written down somewhere other than a code comment.
1. Cherokee confusables are missing
The lookalike table covers Cyrillic, Greek, Armenian, mathematical alphanumerics and fullwidth forms.
Cherokee is used by substitution tools and was left out on purpose rather than guessed at — a
wrong mapping in this table is worse than a missing one, because it produces a false statement about
someone's file. Adding it needs the codepoints checked against the standard, not recalled.
Same for any other script someone can verify: the table is one dictionary in
src/SignsOfAI.Core/Artifacts/Characters.cs.2. PDF text extraction may not see what the reader sees
There is published work on attacking detectors through the PDF layer alone
(arXiv 2508.01887): the extracted text can differ from the
rendered text without either being modified. This touches
SignsOfAI.Documents, which reads PDFon the desktop.
Worth checking what PdfPig does with invisible text layers, off-page text, and fonts with remapped
encodings — and whether the scanner should say anything when extraction and rendering disagree. Not
investigated yet.
3. Non-ASCII Latin lookalikes are only partly covered
ı,ɑ,ɡ,ɩare in the table. Small-capital and other Latin Extended forms are not. These areharder than the cross-script cases because they are Latin, so the mixed-script signal does not
help — the majority-ASCII rule inside a letter run is doing all the work.
4. A false positive is the report I most want
The tool has no server and no telemetry, so a human telling us is the only way we ever find out. If
a document gets flagged and shouldn't be, please open it under False positives in Discussions
with the codepoint and enough context to reproduce.