Repository navigation
Review: English part-of-speech lexicon derived from the bundled Hunspell dictionary #428
Description
Problem
FluentTyper's English Review checks only know whether a word is spelled correctly (Presage dictionary). They don't know what kind of word it is (noun, verb, adjective…) or its forms. So every grammar check is a hand-written regex frame over small authored word lists, and anything outside the lists is silently skipped. Precision is high, but recall is low.
After the Harper-inspired work (docs/harper-parity-evaluation.md), Review catches 1,016 of Harper's 5,907 English grammar cases (exact fix: 857). Most of the remaining misses need part-of-speech (POS) data:
- "He like pizza." → "likes": needs to know
likeis a base-form verb, and its he/she form. - "I need some advise." → "advice": needs to know
adviseis verb-only andadviceis its noun. - "Me and Alex went home." → "Alex and I": needs to know the phrase is the subject of a verb (vs. "They invited me and Alex", which is correct).
- Noun/verb confusions in general, "worth to do", "let him to do", …
Idea: derive a POS lexicon from the Hunspell dictionary we already ship
No new data source, no new license question. Every language except Arabic ships Hunspell and Aspell under resources_js/ (Arabic: Hunspell only). None of them contain explicit morphology (po:/st: fields: 0 in all .dic files). But the English Hunspell dictionary (resources_js/en_US/hunspell/en_US.dic, 49,568 entries, SCOWL-based) marks each word with the affixes it accepts, and those imply the word class:
| Flag | Adds | Implies |
|---|---|---|
G |
-ing | verb |
D |
-ed | verb |
M |
's | noun |
R / T |
-er / -est | adjective |
S |
-s | plural noun or he/she verb |
Y |
-ly | adjective with adverb form |
Examples from the real file:
advise/LDRSZGB→ -ed, -ing, no 's → verbadvice/M→ 's only → nounlike/EMGDST→ verb and noun → ambiguous (must abstain or need context)
A quick count of lowercase alphabetic entries: ~20,000 noun-only, ~2,800 verb-only, ~3,200 verb+noun, ~1,000 adjective-ish.
Aspell adds nothing new for English: same SCOWL word list, but compiled binary (.rws). Hunspell's plain-text .dic + .aff is the easier source.
What it gives / doesn't give
| Case | From Hunspell flags? |
|---|---|
| "some advise" → advice | ✅ advise is verb-only, advice exists as noun |
| "He like" → likes | |
| "Me and Alex went" | |
| Function words (pronouns, prepositions, determiners, conjunctions) | ❌ not marked; ~300 words, easy to author |
| Irregular forms (went, children) | ❌ but already covered by EnglishVerbForms.ts / EnglishNounNumber.ts |
| Context (is "like" a verb here?) | ❌ needs guarded rules or, later, a tagger |
| Noise | spoil looks verb-only (also noun "the spoils"), animate verb-only (also adjective) |
| Other languages | ❌ their flags encode inflection classes, not word class, and every dictionary uses its own scheme (e.g. Polish 300k entries). Possible later, per language. |
Proposed plan (not started)
- Build script (under
scripts/): readen_US.dic+en_US.aff, expand flags into a compact table: word → can be noun / verb / adjective, plus plural and verb forms. Merge in our authored irregular tables and a hand-authored list of function words. Output one generated file (~150–300 KB estimated). - Runtime placement: load it where Review detectors run (pure domain lookup), or in the background next to Presage and query it async. Respect bundle size and the rule that no Local AI / heavy runtime lands in the content script.
- First 2–3 rules using predicates like
isVerbBase(w)/isNounOnly(w)instead of word lists: "some advise", "He like", "me and X". Everything individual-only (never Fix all); ambiguous words abstain. - Measure with the local Harper harness used for the parity doc: detected / exact / false positives on no-lint cases, plus the Markdown false-positive sweep, before and after.
- Later, optional: a small context tagger to disambiguate words like "like", and per-language lexicons.
Costs / risks
- Bundle size: a few hundred KB more.
- English only at first.
- More false positives from ambiguous words → keep every finding a user-accepted proposal, abstain on ambiguity.
- Effort: medium.
Context: follow-up to the Harper-inspired Review work (branch harper-inspired-rules, architecture proposal #11 in docs/harper-parity-evaluation.md).
Activity
bartekplus commentedon Oct 4, 2026
Status check on
master(c8324ac). The plan is implemented. This issue can close.scripts/generate-english-lexicon.tsreadsen_US.dic+.aff, merges the authored irregular verbs and mass nouns, and uses the Presage n-gram counts for count nouns. Output:englishLexicon.generated.ts, about 160 KB, with a drift test.englishWordInfo,hasVerbForm, …) is now used by about 70 modules.docs/languagetool-parity-evaluation.md). The Harper parity doc was removed in #456.Known gap: "He like pizza." gives no finding. "like" before a bare noun abstains, because "like" can also be a preposition. "He like the pizza." and "He like me." are caught. I continue this in a follow-up, with guards, and with no Fix all when the context is not clear.
Implemented in #446 and #456.
Review: catch "He like pizza" as a choice, never in bulk (#460)