Ucdn::$ot_languages (src/Ucdn.php:423, upstream the same table read by Otl::_getOTLLangTag() at src/Otl.php:6150) is a hand-kept subset of the language→language system mapping, and it has drifted from the table HarfBuzz generates (hb-ot-tag-table.hh: ot_languages2, ot_languages3, ot_languages3_multi, ot_languages3_blocked). #206 brings OtlTags::language()'s subtag handling in line with HarfBuzz 14.3.1, but deliberately leaves the table's existing entries alone, because syncing it changes the language system of existing documents.
I compared each of the 354 single-subtag keys the table had before #206 against HarfBuzz 14.3.1's tables, applying HarfBuzz's rule that a three-letter code it has no entry for is used upper-cased as its own tag:
|
keys |
| same first tag |
338 |
| same first tag, but HarfBuzz lists more tags to fall back to |
29 of those 338 |
| first tag differs in HarfBuzz's table |
4 |
| no entry in HarfBuzz's table, and its ISO 639-3 fallback differs |
12 |
in HarfBuzz's tables, not in Ucdn |
35 two-letter, 772 three-letter |
(After #206 adds the Chinese languages, zh, ga → IRI, IRT and nv → NAV, ATH, the last row is 33 and 753 and the second is 28.)
First tag differs (mPDF → HarfBuzz):
ml MLR → MAL, MLR
hy HYE → HYE0, HYE
ber BER → BBR
scs SLA → SCS, SLA, ATH
HarfBuzz's ISO 639-3 fallback differs: grc PGR → GRC, yid JII → YID, nso SOT → NSO, lua LUB → LUA, eot BTI → EOT, kvd KUI → KVD, mdc MLE → MDC, nco SIB → NCO, ril RIA → RIL, umb MBN → UMB, xom KMO → XOM, yso NIS → YSO.
More tags in HarfBuzz, which only change the result where a font lacks the first: e.g. mo → MOL, ROM; mnw → MON, MONT (hb-shape --language=mnw picks MONT from a font offering only that; mPDF uses the default); id → IND, MLY; ko → KOR, KOH; dv → DIV, DHV; iu → INU, INUK; tw → TWI, AKA; crj, crl, crm, cwd → their Cree tag, then YCR/LCR/TCR, then CRE.
Missing keys include two-letter ak, an, co, ia, in, iw, ji, qu, sc, sh, su, tl, za. These currently lay text out with the script's DFLT entry.
HarfBuzz also returns no tag for the 162 three-letter codes in ot_languages3_blocked and upper-cases any other code it does not list. Ucdn has neither rule. That matters for extended language subtags as well: after #206, one Ucdn has no key for keeps the language subtag's tags (ar-afb → ARA), which matches HarfBuzz for most of them.
Expected: generate the table from HarfBuzz's, the way utils/ucdn_update.php generates the Unicode tables, and follow HarfBuzz for these keys. The first-tag changes above alter existing documents, so the PR should list them.
Ucdn::$ot_languages(src/Ucdn.php:423, upstream the same table read byOtl::_getOTLLangTag()atsrc/Otl.php:6150) is a hand-kept subset of the language→language system mapping, and it has drifted from the table HarfBuzz generates (hb-ot-tag-table.hh:ot_languages2,ot_languages3,ot_languages3_multi,ot_languages3_blocked). #206 bringsOtlTags::language()'s subtag handling in line with HarfBuzz 14.3.1, but deliberately leaves the table's existing entries alone, because syncing it changes the language system of existing documents.I compared each of the 354 single-subtag keys the table had before #206 against HarfBuzz 14.3.1's tables, applying HarfBuzz's rule that a three-letter code it has no entry for is used upper-cased as its own tag:
Ucdn(After #206 adds the Chinese languages,
zh,ga→IRI,IRTandnv→NAV,ATH, the last row is 33 and 753 and the second is 28.)First tag differs (mPDF → HarfBuzz):
mlMLR→MAL,MLRhyHYE→HYE0,HYEberBER→BBRscsSLA→SCS,SLA,ATHHarfBuzz's ISO 639-3 fallback differs:
grcPGR→GRC,yidJII→YID,nsoSOT→NSO,luaLUB→LUA,eotBTI→EOT,kvdKUI→KVD,mdcMLE→MDC,ncoSIB→NCO,rilRIA→RIL,umbMBN→UMB,xomKMO→XOM,ysoNIS→YSO.More tags in HarfBuzz, which only change the result where a font lacks the first: e.g.
mo→MOL,ROM;mnw→MON,MONT(hb-shape --language=mnwpicksMONTfrom a font offering only that; mPDF uses the default);id→IND,MLY;ko→KOR,KOH;dv→DIV,DHV;iu→INU,INUK;tw→TWI,AKA;crj,crl,crm,cwd→ their Cree tag, thenYCR/LCR/TCR, thenCRE.Missing keys include two-letter
ak,an,co,ia,in,iw,ji,qu,sc,sh,su,tl,za. These currently lay text out with the script'sDFLTentry.HarfBuzz also returns no tag for the 162 three-letter codes in
ot_languages3_blockedand upper-cases any other code it does not list.Ucdnhas neither rule. That matters for extended language subtags as well: after #206, oneUcdnhas no key for keeps the language subtag's tags (ar-afb→ARA), which matches HarfBuzz for most of them.Expected: generate the table from HarfBuzz's, the way
utils/ucdn_update.phpgenerates the Unicode tables, and follow HarfBuzz for these keys. The first-tag changes above alter existing documents, so the PR should list them.