Select HarfBuzz's language system for a variant, script, region or extended language subtag, and for Chinese other than zh (#206)

[jakejackson1]

Closes #206.

What changes

OtlTags::language() looked a tag up by its first subtag, and handled Chinese separately (#201). HarfBuzz 14.3.1 also reads variants, scripts, regions, extended language subtags, and languages with more than one tag. mPDF now builds its candidates in HarfBuzz's order, checks each against what the script offers, and falls back to DFLT as before.

The order it implements, from hb_ot_tags_from_language() (hb-ot-tag.cc) and hb_ot_tags_from_complex_language() (hb-ot-tag-table.hh):

1. A variant or script that has its own tag in any language, tried in HarfBuzz's order: -fonnapa APPH, -polyton PGR, -arevmda HYE, -provenc PRO, -fonipa IPPH, -geok KGE, -syre/-syrj/-syrn SYRE/SYRJ/SYRN. HarfBuzz's subtag_matches (p, limit, "-polyton", 8) searches every subtag after the language.
2. A retired tag, read whole (strcmp): art-lojban, i-hak, i-lux, i-navajo (NAV, ATH), no-bok, no-nyn, zh-min, zh-min-nan. Without these, step 5 would read no-nyn as Nkole.
3. A script or region with its own tag in that language: ga-Latg IRT (lang_matches, so the script must come straight after the language), mnw-TH MONT, ro-MD MOL then ROM (subtag_matches, so the region can be anywhere).
4. Chinese scripts and regions for every Chinese language (below).
5. An extended language subtag: "If there is an extended language tag, use it", meaning a three-letter second subtag starting with a letter. So zh-yue gives ZHH. This runs after step 4, so zh-yue-HK gives ZHH by its region and zh-yue-Hans gives ZHH by yue, as in HarfBuzz.
6. The language subtag, trying each of its tags in turn (ot_languages2 lists ga → IRI, IRT and nv → NAV, ATH).

As in hb_ot_tags_from_script_and_language(), subtags after the first singleton (-x-, -u-, …) are ignored, and a tag starting with x- gives nothing.

Chinese

chinese() now takes the language's own tag rather than assuming zh. Ucdn::$ot_languages gains zh → ZHS and the 19 other Chinese languages HarfBuzz lists: cdo cjy cmn cnp cpx csp czh czo gan hak hnm hsn luh mnp nan sjc wuu → ZHS, yue → ZHH, lzh → ZHT. A language whose own tag is ZHS, ZHT or ZHH counts as Chinese. The rules follow HarfBuzz's generator (gen-tag-table.py removes yue and lzh from the zh macrolanguage and gives each only a -Hans rule):

• -Hans gives ZHS for every Chinese language.
• zh and its ZHS members take all of zh's rules: Hant-HK ZHH, Hant-MO ZHTM then ZHH, Hant ZHT, then region hk ZHH, mo ZHTM then ZHH, tw ZHT.
• yue and lzh keep their own tag for any other script or region: yue-Hant-TW and yue-MO give ZHH, and lzh-HK gives ZHT. hb-shape confirms this. The issue said every Chinese language takes zh's rules, but HarfBuzz doesn't do that for these two.

Ucdn::$ot_languages

• Added zh and the Chinese languages above.
• ga is now ['IRI ', 'IRT '], and nv (which had no entry) is ['NAV ', 'ATH ']. A value can now be an array. OtlTags is the only reader in this fork, and Otl::_getOTLLangTag() is the only one upstream (checked with gh search code). Any third-party code reading the public static would see an array for these two keys.
• Removed the five zh-cn, zh-hk, zh-mo, zh-sg, zh-tw keys. Nothing has read them since A Chinese script subtag never selects ZHS or ZHT, and bare zh and zh-MO pick different language systems from HarfBuzz #201, zh-mo's ZHT was wrong, and chinese() now owns every Chinese mapping.

Where mPDF still differs from HarfBuzz

When HarfBuzz has no table entry for a three-letter code, it upper-cases the code and uses that as the tag. mPDF doesn't do this. So an extended language that Ucdn has no key for keeps the language subtag's tags: ar-afb stays ARA. That matches HarfBuzz for most extended languages, because its table gives them their macrolanguage's tag. Otherwise, adding step 5 would have turned ar-afb from ARA into DFLT. The same table gap means bare mnw still gives MON, where HarfBuzz gives MON then MONT. Both belong to #211.

Scope: hand-coded rules, not a full table sync

Before choosing, I compared all 354 single-subtag keys in Ucdn::$ot_languages with HarfBuzz 14.3.1's ot_languages2/ot_languages3/ot_languages3_multi, including its ISO 639-3 fallback:

	keys
same first tag	338
…of which HarfBuzz lists more tags	29
first tag differs in HarfBuzz's table (ml MLR→MAL, hy HYE→HYE0, ber BER→BBR, scs SLA→SCS)	4
not in HarfBuzz's tables, and its ISO fallback differs (grc PGR→GRC, yid JII→YID, nso SOT→NSO, …)	12
in HarfBuzz, missing from Ucdn	35 two-letter, 772 three-letter

A sync would change the first tag for 16 existing keys, and some of them (Malayalam, Armenian, Ancient Greek, Yiddish) affect real documents. It would also add about 800 keys. That's a different kind of change from this issue, so this PR hand-codes the issue's cases and their mechanisms, and #211 tracks the sync with these numbers and examples.

As a check that the mechanism matches HarfBuzz once the data does, I swapped HarfBuzz's own table into Ucdn::$ot_languages at runtime. I then compared the candidates for 87,094 generated tags (36 languages, crossed with scripts, regions, variants, extended languages and singletons) against hb_ot_tags_from_script_and_language() called through libharfbuzz 14.3.1. The only differences were HarfBuzz's ISO 639-3 fallback (deliberately not implemented) and 99 malformed i-<subtag> tags, which HarfBuzz rejects through a length check.

Behaviour change

• Multi-tag languages: a font offering IRT but not IRI now uses IRT for lang="ga". Before, it used DFLT. lang="nv" now selects NAV, or ATH if NAV isn't offered.
• Fonts in this repo: no font in tests/data/ttf or packages/*/fonts offers IRT, ATH, PGR, PRO, KGE, SYRE/J/N, MONT or HYE0. The ones that offer an affected tag:
  • NotoSans-Regular and NotoSansMono-GDEF13-Subset offer NAV under latn, so lang="nv" now uses it instead of DFLT.
  • NotoSans-Regular, NotoSansMono-GDEF13-Subset and Carlito-MarkAttachmentType-Subset offer IPPH, and the two Noto fonts also offer APPH. -fonipa and -fonnapa now select them.
  • DejaVu Sans and Serif offer both MOL and ROM under latn, and ro-MD now uses MOL instead of ROM.
  • The synthetic Chinese test fonts are unaffected apart from what A Chinese script subtag never selects ZHS or ZHT, and bare zh and zh-MO pick different language systems from HarfBuzz #201 already covers.
• Extended languages: after this change, the only extended languages that Ucdn has keys for are the Chinese ones. So the extended-language rule changes nothing outside zh-*.

Test font and tests

Noto-LanguageTags-Synthetic.ttf (4 KB) merges subsets of Noto Sans TC 2.004, Noto Sans Georgian 2.005, Noto Sans Myanmar 2.107 and Noto Sans Syriac 3.000 (all OFL 1.1) using fontTools 4.59.2, and replaces the GSUB. No script has features by default. Each language system has one locl lookup that substitutes a different glyph:

• latn: a → h under IPPH, i under IRT, m under MOL, p under PRO, t under ATH
• grek: α → β under PGR
• geor: Ⴀ → Ⴁ under KGE
• mym2: က → ခ under MONT
• syrc: ܐ → ܒ under SYRE
• hani: 骨 → 一 under ZHS, 二 under ZHT, 三 under ZHH

No script offers IRI, ROM or NAV, so ga and nv show the fallback to the second tag.

LanguageTagLangSysTest checks the glyph mPDF draws (via TextRecordingMpdf) for every row of the issue's table and a few more. Each result was confirmed with hb-shape --language=… --unicodes=…:

lang	text	before	after	hb-shape
el-polyton	α	α	β (PGR)	β
ga-Latg	a	a	i (IRT)	i
ro-MD	a	a	m (MOL)	m
mnw-TH	က	က	ခ (MONT)	ခ
oc-provenc	a	a	p (PRO)	p
syr-Syre	ܐ	ܐ	ܒ (SYRE)	ܒ
en-fonipa	a	a	h (IPPH)	h
ka-Geok	Ⴀ	Ⴀ	Ⴁ (KGE)	Ⴁ
yue	骨	骨	三 (ZHH)	三
yue-Hant-HK	骨	骨	三 (ZHH)	三
cmn-Hans	骨	骨	一 (ZHS)	一
lzh	骨	骨	二 (ZHT)	二
zh-yue	骨	一 (ZHS)	三 (ZHH)	三
zh-lzh	骨	一 (ZHS)	二 (ZHT)	二
ga	a	a	i (IRT after IRI)	i
nv	a	a	t (ATH after NAV)	t
cmn-Hant-TW	骨	骨	二 (ZHT)	二
yue-Hans	骨	骨	一 (ZHS)	一
ro	a	a	a (ROM not offered)	a

OtlTagsTest adds:

• every row of the issue's table, with every tag offered
• positional cases: a variant after a region, a region after a script, upper case, a variant after -x- or -u-, and x- alone
• the Chinese generalisation: yue-Hant-TW, yue-MO, lzh-HK, lzh-Hans, cmn-MO, hak-HK, nan
• extended languages: zh-yue-HK, zh-yue-Hans, zh-cmn-Hant, and ar-afb keeping ARA
• the retired tags
• fallbacks: ro-MD with MOL → MOL; without MOL → ROM; with neither → DFLT. The same for ga with IRI/IRT, plus nv → ATH and ga-Latg without IRT → DFLT

I ran every expectation (80 of them) through hb_ot_tags_from_script_and_language() against the same offered tags, and all match.

Mutation check, one mechanism at a time, running OtlTagsTest, LanguageTagLangSysTest and ChineseLangSysTest:

mutation	fails
variant rule skipped	11: drawn el-polyton, oc-provenc, syr-Syre, en-fonipa, ka-Geok; unit tests for the same five plus el-GR-polyton
extended-language rule skipped	5: drawn zh-yue, zh-lzh; unit zh-yue, zh-lzh, zh-yue-Hans
only a language's first tag	4: drawn ga, nv; fallbacks "Irish, without IRI", "Navajo, without NAV"
Chinese rules only for zh	7: drawn cmn-Hant-TW, yue-Hans; unit yue-Hans, lzh-Hans, cmn-Hant-TW, cmn-MO, hak-HK
zh's rules applied to yue/lzh too	3: yue-Hant-TW, yue-MO, lzh-HK
ga/mnw/ro rules skipped	9, including drawn ro-MD and mnw-TH
retired tags skipped	4: i-navajo, no-nyn, i-lux, art-lojban
singleton cut skipped	2: ro-x-md, el-u-polyton

Fixtures

I did a cold regeneration: deleted tests/Mpdf/tmp/mpdf, tmp/mpdf and tmp/ttfontdata, then ran composer fontcache:update all, otldump:update all, shaping:update all and subset:update all. The only files written were the four new ones for Noto-LanguageTags-Synthetic, and no existing fixture moved. The new otldump lists the six scripts with one locl per language system. The shaping fixture shows no substitution, because the golden master sets no language.

Verification

• Full suite: 2188 tests, 6079 assertions, 1 skipped (baseline 2110 / 5991 / 1).
• Snapshots: 79 / 110, OK.
• composer cs clean.
• phpstan: 33, none in OtlTags.php or Ucdn.php.

Simplify pass

After the review, the second commit simplifies languageSystems():

• the singleton cut is now one preg_replace
• the language is array_shifted off, so a single subtag array serves every rule
• chinese() now decides for itself whether the language is Chinese
• the extended language is looked up once
• an empty tag returns early

Tests, the HarfBuzz comparison and every mutation give the same results as before.

Suggestions I didn't take:

• A per-tag result cache: Ucdn::$ot_languages is public and can be changed, so a cache could go stale, and the call costs microseconds.
• Moving the drawn-glyph helper into a shared trait: there are already six copies across tests, so it's a change of its own.
• Deriving $retired from language subtags: half of the entries would still need literals.
• Parsing the tag into BCP 47 fields and a single generated rule list, making every table value an array, and naming the Chinese languages explicitly: these belong with the table sync. I noted them on Ucdn::$ot_languages has drifted from HarfBuzz's language tag table: 16 first tags differ and 800 languages are missing, e.g. ml, grc, qu #211.

Stack

This PR's base is fix/201-chinese-script-subtags (#207), and the stack continues #207 → #202 → #200 → #198. #190, #188 and #186 have been squash-merged into gravitypdf. #207 is still open, so this branch is not rebased.

Upstream mirror

Upstream, the equivalent is Otl::_getOTLLangTag() (src/Otl.php:6150 on mpdf/mpdf development) plus Ucdn::$ot_languages. The mirror moves languageSystems(), chinese(), tags() and the $variants, $retired, $languageSubtags and $chineseRegions tables into Otl as private members, and makes the same Ucdn edits. It depends on the mirror of #191/#201, which introduced the ordered-candidates loop and chinese(). Apply it after that one.

🤖 Generated with Claude Code

[jakejackson1]

Rebased: #190 was squash-merged into gravitypdf (f9b3c0e), so this stack still carried its pre-squash commits. This branch is rebased onto its rebased parent with --onto; no conflicts and the base is unchanged. Suite, snapshots and cs pass on the new head, with no fixture movement.

[jakejackson1]

This PR merged into a base branch that had already been squash-merged, so its change never reached gravitypdf. It's carried to gravitypdf by #221.