Skip to content

Ship hi and bn locale packs for Latin-transliterated Indic honorifics #345

Open

Description

@derek73

#343 and #344 add native-script Indic honorifics to the default vocabulary, on #269's reasoning that native-script entries cannot collide with Latin-script names. The transliterated forms — which is how most Western-sourced datasets store these names — have no route into the default set, because they collide exactly as #269 said:

Shri Narendra Modi       -> given='Shri'   family='Modi'      ❌
Pandit Ravi Shankar      -> given='Pandit' family='Shankar'   ❌
Sardar Vallabhbhai Patel -> given='Sardar' family='Patel'     ❌

Also unrecognized: smt, shrimati, kumari, sushri, shriyut, babu, acharya, janab, begum. (md already ships as a title, so Md Abdul Karim parses correctly today — no change needed.)

A locale pack is the right home: it scopes the collision to the datasets whose owners know it is the right reading.

Prototype — verified working at 2.1.0

HI = Locale(code="hi", lexicon=Lexicon(titles=frozenset(
    {"shri","sri","smt","shrimati","pandit","pt","sardar",
     "kumari","sushri","shriyut","babu","acharya"})))
BN = Locale(code="bn", lexicon=Lexicon(titles=frozenset(
    {"janab","begum","shri","sri","shrimati","adhyapak"})))

p = parser_for(HI, BN)
Shri Narendra Modi        -> title='Shri'    given='Narendra'  family='Modi'    ✅
Pandit Ravi Shankar       -> title='Pandit'  given='Ravi'      family='Shankar' ✅
Sardar Vallabhbhai Patel  -> title='Sardar'  given='Vallabhbhai' family='Patel' ✅
Janab Abul Kalam          -> title='Janab'   given='Abul'      family='Kalam'   ✅
Begum Khaleda Zia         -> title='Begum'   given='Khaleda'   family='Zia'     ✅
Acharya Vinoba Bhave      -> title='Acharya' given='Vinoba'    family='Bhave'   ✅

And the cost, which stays opt-in:

Sri Mulyani Indrawati   default -> given='Sri'                 ✅
                        pack    -> title='Sri' given='Mulyani' ❌

That is the intended trade — the ambiguity is irreducible, so it is resolved by whoever knows whether the dataset is Indic honorifics or Indonesian given names.

Two packs, not one

parser_for composes left-to-right, so parser_for(locales.HI, locales.BN) works and the vocabularies union — verified above. The lists overlap (shri, sri, shrimati) but diverge meaningfully (pandit/sardar vs janab/begum), and separate codes let a Bangladesh-only dataset avoid the Hindi collisions it does not need. tr_az bundles two languages, but it carries no vocabulary at all — the shared thing there is a mechanism, not a word list.

Notes for implementation

This would be the first title-carrying pack: ru and tr_az are policy-only (Lexicon.empty()), and zh carries surnames. Nothing in the design forbids it — Locale.lexicon unions onto the base — but it is a new shape for the pack layer.

The registry contract (tests/v2/test_locales.py:28) makes adding a pack one _REGISTRY entry plus one rotator list, with a meta-test that fails until both exist.

DEVIATES is straightforward for a vocabulary-only pack — any token in the pack's added vocabulary. Over-declaring is the safe direction, per the ru pack's note:

def DEVIATES(name: str) -> bool:
    return any(t.strip(".").lower() in _WORDS for t in name.split())

Explicit non-goal

These packs will not fix initials-first South Indian names:

P. V. Sindhu -> given='P.' middle='V.' family='Sindhu'

"Sindhu" is the personal name; "P. V." is father/village. That convention varies by person rather than by language, so there is no key — script, locale, or otherwise — to hang it on. It belongs in docs/locales.rst as a stated limit, not in a pack.

Open questions

  • Vocabulary scope per pack. The prototype lists are a starting point, not a vetted set; each entry needs the Provide constants in non-Latin scripts (Cyrillic, Greek, Arabic, Hebrew) #269-style check against real Latin given names and surnames.
  • begum, kumari. Both are also name components (Kumari is a common given name). Inside an opt-in pack the bar is lower than for the default set — but "lower" still is not "none", since the pack's users will have both readings in their data.
  • Marathi/Nepali. Both write Devanagari and share much of the hi vocabulary. Fold in, or separate codes later?

Activity

  1. self-assigned this
    on Oct 11, 2026
  2. added
    vocabularyPer-word vocabulary model and language packs
    on Oct 19, 2026
  3. added this to the 3.0 milestone on Oct 29, 2026
  4. derek73 commented on Oct 31, 2026

    @derek73
    OwnerAuthor

    Status after 3.0's sub-phase 3e (#958): locales.HI and locales.BN ship as opt-in language-choice packs. Both read smt and shrimati; locales.HI also reads shriyut, and locales.BN reads adhyapak. shri was left out on measurement (198 ORCID given names open with it), and sushri waits for strict mode, being an Odia given name. The criterion and counts are in decisions/vocabulary-model's 2026-10-30 phase 3e entry.

    Still open here (#971 narrows the decisions file's Open: line to match): the transliterated honorifics no pack reads yet, pandit, sardar, kumari, babu, acharya, janab, begum, plus shri and sushri.

  5. added a commit that references this issue on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementvocabularyPer-word vocabulary model and language packs

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions