Skip to content

Ucdn::$ot_languages has drifted from HarfBuzz's language tag table: 16 first tags differ and 800 languages are missing, e.g. ml, grc, qu #211

Description

@jakejackson1

Ucdn::$ot_languages (src/Ucdn.php:423, upstream the same table read by Otl::_getOTLLangTag() at src/Otl.php:6150) is a hand-kept subset of the language→language system mapping, and it has drifted from the table HarfBuzz generates (hb-ot-tag-table.hh: ot_languages2, ot_languages3, ot_languages3_multi, ot_languages3_blocked). #206 brings OtlTags::language()'s subtag handling in line with HarfBuzz 14.3.1, but deliberately leaves the table's existing entries alone, because syncing it changes the language system of existing documents.

I compared each of the 354 single-subtag keys the table had before #206 against HarfBuzz 14.3.1's tables, applying HarfBuzz's rule that a three-letter code it has no entry for is used upper-cased as its own tag:

keys
same first tag 338
same first tag, but HarfBuzz lists more tags to fall back to 29 of those 338
first tag differs in HarfBuzz's table 4
no entry in HarfBuzz's table, and its ISO 639-3 fallback differs 12
in HarfBuzz's tables, not in Ucdn 35 two-letter, 772 three-letter

(After #206 adds the Chinese languages, zh, gaIRI, IRT and nvNAV, ATH, the last row is 33 and 753 and the second is 28.)

First tag differs (mPDF → HarfBuzz):

  • ml MLRMAL, MLR
  • hy HYEHYE0, HYE
  • ber BERBBR
  • scs SLASCS, SLA, ATH

HarfBuzz's ISO 639-3 fallback differs: grc PGRGRC, yid JIIYID, nso SOTNSO, lua LUBLUA, eot BTIEOT, kvd KUIKVD, mdc MLEMDC, nco SIBNCO, ril RIARIL, umb MBNUMB, xom KMOXOM, yso NISYSO.

More tags in HarfBuzz, which only change the result where a font lacks the first: e.g. moMOL, ROM; mnwMON, MONT (hb-shape --language=mnw picks MONT from a font offering only that; mPDF uses the default); idIND, MLY; koKOR, KOH; dvDIV, DHV; iuINU, INUK; twTWI, AKA; crj, crl, crm, cwd → their Cree tag, then YCR/LCR/TCR, then CRE.

Missing keys include two-letter ak, an, co, ia, in, iw, ji, qu, sc, sh, su, tl, za. These currently lay text out with the script's DFLT entry.

HarfBuzz also returns no tag for the 162 three-letter codes in ot_languages3_blocked and upper-cases any other code it does not list. Ucdn has neither rule. That matters for extended language subtags as well: after #206, one Ucdn has no key for keeps the language subtag's tags (ar-afbARA), which matches HarfBuzz for most of them.

Expected: generate the table from HarfBuzz's, the way utils/ucdn_update.php generates the Unicode tables, and follow HarfBuzz for these keys. The first-tag changes above alter existing documents, so the PR should list them.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions