Skip to content

Select HarfBuzz's language system for a variant, script, region or extended language subtag, and for Chinese other than zh (#206) #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jakejackson1 merged 2 commits into from
Sep 17, 2026
Merged

Select HarfBuzz's language system for a variant, script, region or extended language subtag, and for Chinese other than zh (#206) #213

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
85ec38e
Select HarfBuzz's language system for a variant, script, region or ex…
jakejackson1 Sep 17, 2026
86f0d6e
Cut a language tag at its first singleton in one step, and ask chines…
jakejackson1 Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 135 additions & 21 deletions src/Shaper/OtlTags.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,46 @@ class OtlTags
'mym2' => 'mymr',
];

/**
* Variant and script subtags with a language system of their own whatever the language, in the
* order HarfBuzz tests them.
*/
private static $variants = [
'fonnapa' => ['APPH'],
'polyton' => ['PGR '],
'arevmda' => ['HYE '],
'provenc' => ['PRO '],
'fonipa' => ['IPPH'],
'geok' => ['KGE '],
'syre' => ['SYRE'],
'syrj' => ['SYRJ'],
'syrn' => ['SYRN'],
];

/**
* Retired tags HarfBuzz reads whole, before the extended language rule would read no-nyn as
* Nkole.
*/
private static $retired = [
'art-lojban' => ['JBO '],
'i-hak' => ['ZHS '],
'i-lux' => ['LTZ '],
'i-navajo' => ['NAV ', 'ATH '],
'no-bok' => ['NOR '],
'no-nyn' => ['NYN '],
'zh-min' => ['ZHS '],
'zh-min-nan' => ['ZHS '],
];

/**
* Languages that take another language system for one script or region.
*/
private static $languageSubtags = [
'ga' => ['latg' => ['IRT ']],
'mnw' => ['th' => ['MONT']],
'ro' => ['md' => ['MOL ', 'ROM ']],
];

/**
* The Chinese regions with language systems of their own, in the order HarfBuzz looks for them.
*/
Expand Down Expand Up @@ -95,48 +135,112 @@ public static function language($ietf, $available)
return '';
}

$subtags = $ietf ? explode('-', strtolower($ietf)) : [];
foreach (self::languageSystems($ietf) as $langsys) {
if (strpos($available, $langsys) !== false) {
return $langsys;
}
}

return strpos($available, 'DFLT') !== false ? 'DFLT' : '';
}

/**
* The language systems for an IETF tag, in the order HarfBuzz 14.3.1 tries them
* (hb_ot_tags_from_language() in hb-ot-tag.cc and hb_ot_tags_from_complex_language() in
* hb-ot-tag-table.hh):
*
* 1. a variant or script that has its own language system in any language, e.g. -polyton
* 2. a retired tag, read whole
* 3. a script or region with its own language system in this language, e.g. ro-MD
* 4. Chinese scripts and regions
* 5. an extended language subtag, e.g. zh-yue
* 6. the language subtag
*
* Only the subtags before the first singleton count, so an extension or private use subtag
* selects nothing. A script is the subtag after the language; a region may be anywhere.
*
* An extended language Ucdn::$ot_languages has no key for keeps the language subtag's tags, where
* HarfBuzz would take an unlisted one as its own ISO 639-3 code.
*
* @param string $ietf
*
* @return string[]
*/
private static function languageSystems($ietf)
{
$tag = strtolower((string) $ietf);
if ($tag == '') {
return [];
}

$candidates = [];
if (isset($subtags[0]) && $subtags[0] == 'zh') {
$candidates = self::chinese($subtags);
} elseif (isset($subtags[0]) && isset(Ucdn::$ot_languages[$subtags[0]])) {
$candidates = [Ucdn::$ot_languages[$subtags[0]]];
$subtags = explode('-', preg_replace('/-[^-]-.*/s', '', $tag));
$language = array_shift($subtags);
if ($language == 'x') {
return [];
}

foreach ($candidates as $langsys) {
if (strpos($available, $langsys) !== false) {
foreach (self::$variants as $variant => $langsys) {
if (in_array($variant, $subtags, true)) {
return $langsys;
}
}

return strpos($available, 'DFLT') !== false ? 'DFLT' : '';
if (isset(self::$retired[$tag])) {
return self::$retired[$tag];
}

$script = isset($subtags[0]) ? $subtags[0] : '';

if (isset(self::$languageSubtags[$language])) {
foreach (self::$languageSubtags[$language] as $subtag => $langsys) {
if (strlen($subtag) == 4 ? $script == $subtag : in_array($subtag, $subtags, true)) {
return $langsys;
}
}
}

$own = self::tags($language);

$chinese = self::chinese($own, $script, $subtags);
if ($chinese) {
return $chinese;
}

// A three-digit region such as 419 has no key, so three characters that find one are a language
$extended = strlen($script) == 3 ? self::tags($script) : [];

return $extended ?: $own;
}

/**
* The language systems for Chinese, in the order HarfBuzz tries them
* (hb_ot_tags_from_complex_language() in hb-ot-tag-table.hh).
* The Chinese language systems a script or region gives, in the order HarfBuzz tries them.
*
* The script subtag outranks the region, so zh-Hans-HK is Simplified, except that Traditional
* Chinese in Hong Kong or Macao keeps its region's tag. Macao has its own ZHTM, and where a font
* lacks it takes Hong Kong's ZHH before the default. A tag with neither is Simplified.
* lacks it takes Hong Kong's ZHH before the default.
*
* The zh- keys of Ucdn::$ot_languages are not read: they hold one tag per region, and nothing for
* the script or for zh alone.
* Only Hans moves Cantonese (yue, ZHH) and Literary Chinese (lzh, ZHT) off their own tags. Every
* other Chinese language is Simplified by default and takes zh's rules.
*
* @param string[] $subtags The lower-cased subtags of a tag whose language is zh
* @param string[] $own The language's own tags
* @param string $script The subtag after the language
* @param string[] $subtags The subtags after the language
*
* @return string[]
* @return string[] The language systems, or none
*/
private static function chinese(array $subtags)
private static function chinese(array $own, $script, array $subtags)
{
$script = isset($subtags[1]) ? $subtags[1] : '';

if (!$own || !in_array($own[0], ['ZHS ', 'ZHT ', 'ZHH '], true)) {
return [];
}
if ($script == 'hans') {
return ['ZHS '];
}
if ($own[0] != 'ZHS ') {
return [];
}
if ($script == 'hant') {
$region = isset($subtags[2]) ? $subtags[2] : '';
$region = isset($subtags[1]) ? $subtags[1] : '';

return $region == 'hk' || $region == 'mo' ? self::$chineseRegions[$region] : ['ZHT '];
}
Expand All @@ -147,7 +251,17 @@ private static function chinese(array $subtags)
}
}

return ['ZHS '];
return [];
}

/**
* @param string $language A language subtag
*
* @return string[] Its language systems in Ucdn::$ot_languages, or none
*/
private static function tags($language)
{
return isset(Ucdn::$ot_languages[$language]) ? (array) Ucdn::$ot_languages[$language] : [];
}

/**
Expand Down
28 changes: 22 additions & 6 deletions src/Ucdn.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,23 +471,31 @@ public static function get_script($code)
'bxr' => 'RBU ', /* Russian Buriat */
'byn' => 'BIL ', /* Bilen */
'ca' => 'CAT ', /* Catalan */
'cdo' => 'ZHS ', /* Min Dong Chinese */
'ce' => 'CHE ', /* Chechen */
'ceb' => 'CEB ', /* Cebuano */
'chp' => 'CHP ', /* Chipewyan */
'chr' => 'CHR ', /* Cherokee */
'cjy' => 'ZHS ', /* Jinyu Chinese */
'ckt' => 'CHK ', /* Chukchi */
'cmn' => 'ZHS ', /* Mandarin Chinese */
'cnp' => 'ZHS ', /* Northern Ping Chinese */
'cop' => 'COP ', /* Coptic */
'cpx' => 'ZHS ', /* Pu-Xian Chinese */
'cr' => 'CRE ', /* Cree */
'crh' => 'CRT ', /* Crimean Tatar */
'crj' => 'ECR ', /* [Southern] East Cree */
'crl' => 'ECR ', /* [Northern] East Cree */
'crm' => 'MCR ', /* Moose Cree */
'crx' => 'CRR ', /* Carrier */
'cs' => 'CSY ', /* Czech */
'csp' => 'ZHS ', /* Southern Ping Chinese */
'cu' => 'CSL ', /* Church Slavic */
'cv' => 'CHU ', /* Chuvash */
'cwd' => 'DCR ', /* Woods Cree */
'cy' => 'WEL ', /* Welsh */
'czh' => 'ZHS ', /* Huizhou Chinese */
'czo' => 'ZHS ', /* Min Zhong Chinese */
'da' => 'DAN ', /* Danish */
'dap' => 'NIS ', /* Nisi (India) */
'dar' => 'DAR ', /* Dargwa */
Expand Down Expand Up @@ -522,9 +530,10 @@ public static function get_script($code)
'fr' => 'FRA ', /* French */
'fur' => 'FRL ', /* Friulian */
'fy' => 'FRI ', /* Western Frisian */
'ga' => 'IRI ', /* Irish */
'ga' => ['IRI ', 'IRT '], /* Irish */
'gaa' => 'GAD ', /* Ga */
'gag' => 'GAG ', /* Gagauz */
'gan' => 'ZHS ', /* Gan Chinese */
'gbm' => 'GAW ', /* Garhwali */
'gd' => 'GAE ', /* Scottish Gaelic */
'gez' => 'GEZ ', /* Ge'ez */
Expand All @@ -538,18 +547,21 @@ public static function get_script($code)
'guk' => 'GMZ ', /* Gumuz */
'gv' => 'MNX ', /* Manx Gaelic */
'ha' => 'HAU ', /* Hausa */
'hak' => 'ZHS ', /* Hakka Chinese */
'har' => 'HRI ', /* Harari */
'haw' => 'HAW ', /* Hawaiin */
'he' => 'IWR ', /* Hebrew */
'hi' => 'HIN ', /* Hindi */
'hil' => 'HIL ', /* Hiligaynon */
'hnd' => 'HND ', /* [Southern] Hindko */
'hne' => 'CHH ', /* Chattisgarhi */
'hnm' => 'ZHS ', /* Hainanese */
'hno' => 'HND ', /* [Northern] Hindko */
'hoc' => 'HO ', /* Ho */
'hoj' => 'HAR ', /* Harauti */
'hr' => 'HRV ', /* Croatian */
'hsb' => 'USB ', /* Upper Sorbian */
'hsn' => 'ZHS ', /* Xiang Chinese */
'ht' => 'HAI ', /* Haitian */
'hu' => 'HUN ', /* Hungarian */
'hy' => 'HYE ', /* Armenian */
Expand Down Expand Up @@ -619,10 +631,12 @@ public static function get_script($code)
'lt' => 'LTH ', /* Lithuanian */
'lu' => 'LUB ', /* Luba-Katanga */
'lua' => 'LUB ', /* Luba-Kasai */
'luh' => 'ZHS ', /* Leizhou Chinese */
'luo' => 'LUO ', /* Luo (Kenya and Tanzania) */
'lus' => 'MIZ ', /* Mizo */
'luy' => 'LUH ', /* Luhya [macrolanguage] */
'lv' => 'LVI ', /* Latvian */
'lzh' => 'ZHT ', /* Literary Chinese */
'lzz' => 'LAZ ', /* Laz */
'mai' => 'MTH ', /* Maithili */
'mdc' => 'MLE ', /* Male (Papua New Guinea) */
Expand All @@ -638,6 +652,7 @@ public static function get_script($code)
'mnc' => 'MCH ', /* Manchu */
'mni' => 'MNI ', /* Manipuri */
'mnk' => 'MND ', /* Mandinka */
'mnp' => 'ZHS ', /* Min Bei Chinese */
'mns' => 'MAN ', /* Mansi */
'mnw' => 'MON ', /* Mon */
'mo' => 'MOL ', /* Moldavian */
Expand All @@ -652,6 +667,7 @@ public static function get_script($code)
'mym' => 'MEN ', /* Me'en */
'myv' => 'ERZ ', /* Erzya */
'nag' => 'NAG ', /* Naga-Assamese */
'nan' => 'ZHS ', /* Min Nan Chinese */
'nb' => 'NOR ', /* Norwegian Bokmål */
'nco' => 'SIB ', /* Sibe */
'nd' => 'NDB ', /* [North] Ndebele */
Expand All @@ -670,6 +686,7 @@ public static function get_script($code)
'nr' => 'NDB ', /* [South] Ndebele */
'nsk' => 'NAS ', /* Naskapi */
'nso' => 'SOT ', /* [Northern] Sotho */
'nv' => ['NAV ', 'ATH '], /* Navajo */
'ny' => 'CHI ', /* Nyanja */
'nyn' => 'NKL ', /* Nkole */
'oc' => 'OCI ', /* Occitan (post 1500) */
Expand Down Expand Up @@ -711,6 +728,7 @@ public static function get_script($code)
'shn' => 'SHN ', /* Shan */
'si' => 'SNH ', /* Sinhala */
'sid' => 'SID ', /* Sidamo */
'sjc' => 'ZHS ', /* Shaojiang Chinese */
'sjd' => 'KSM ', /* Kildin Sami */
'sk' => 'SKY ', /* Slovak */
'skr' => 'SRK ', /* Seraiki */
Expand Down Expand Up @@ -765,6 +783,7 @@ public static function get_script($code)
'wbm' => 'WA ', /* Wa */
'wbr' => 'WAG ', /* Wagdi */
'wo' => 'WLF ', /* Wolof */
'wuu' => 'ZHS ', /* Wu Chinese */
'xal' => 'KLM ', /* Kalmyk */
'xh' => 'XHS ', /* Xhosa */
'xom' => 'KMO ', /* Komo (Sudan) */
Expand All @@ -773,13 +792,10 @@ public static function get_script($code)
'yid' => 'JII ', /* Yiddish */
'yo' => 'YBA ', /* Yoruba */
'yso' => 'NIS ', /* Nisi (China) */
'yue' => 'ZHH ', /* Yue Chinese */
'zh' => 'ZHS ', /* Chinese */
'zne' => 'ZND ', /* Zande */
'zu' => 'ZUL ', /* Zulu */
'zh-cn' => 'ZHS ', /* Chinese (China) */
'zh-hk' => 'ZHH ', /* Chinese (Hong Kong) */
'zh-mo' => 'ZHT ', /* Chinese (Macao) */
'zh-sg' => 'ZHS ', /* Chinese (Singapore) */
'zh-tw' => 'ZHT ', /* Chinese (Taiwan) */
];

// hb-unicode.h
Expand Down
64 changes: 64 additions & 0 deletions tests/Mpdf/LanguageTagLangSysTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
<?php

namespace Mpdf;

/**
* A variant, script, region or extended language subtag, a Chinese language other than zh, and a
* language with more than one tag select the font's language system the way HarfBuzz does.
*
* Noto-LanguageTags-Synthetic merges subsets of Noto Sans TC 2.004, Noto Sans Georgian 2.005, Noto
* Sans Myanmar 2.107 and Noto Sans Syriac 3.000 (all OFL 1.1), with its GSUB replaced. Each script
* has no features by default, and one 'locl' lookup per language system that gives one character
* another's glyph: a becomes h under IPPH, i under IRT, m under MOL, p under PRO and t under ATH;
* α becomes β under PGR, Ⴀ Ⴁ under KGE, က ခ under MONT and ܐ ܒ under SYRE; 骨 becomes 一 under
* ZHS, 二 under ZHT and 三 under ZHH. No script offers IRI, ROM or NAV. `hb-shape --language`
* draws the same glyph for every case below.
*/
class LanguageTagLangSysTest extends \Yoast\PHPUnitPolyfills\TestCases\TestCase
{

/**
* @dataProvider languages
*/
public function testTheLanguageSelectsTheLanguageSystem($lang, $character, $expected)
{
$mpdf = new TextRecordingMpdf([
'mode' => 'utf-8',
'fontDir' => [__DIR__ . '/../data/ttf'],
'fontdata' => ['notolanguagetagssynthetic' => [
'R' => 'Noto-LanguageTags-Synthetic.ttf',
'useOTL' => 0xFF,
]],
'default_font' => 'notolanguagetagssynthetic',
]);
$mpdf->WriteHTML(sprintf('<p lang="%s">&#x%04X;</p>', $lang, $character));

$this->assertSame([$expected], array_values(unpack('N*', mb_convert_encoding($mpdf->drawnText[0], 'UTF-32BE', 'UTF-8'))));
}

public function languages()
{
return [
'el-polyton, PGR' => ['el-polyton', 0x03B1, 0x03B2],
'ga-Latg, IRT' => ['ga-Latg', 0x61, 0x69],
'ro-MD, MOL' => ['ro-MD', 0x61, 0x6D],
'mnw-TH, MONT' => ['mnw-TH', 0x1000, 0x1001],
'oc-provenc, PRO' => ['oc-provenc', 0x61, 0x70],
'syr-Syre, SYRE' => ['syr-Syre', 0x0710, 0x0712],
'en-fonipa, IPPH' => ['en-fonipa', 0x61, 0x68],
'ka-Geok, KGE' => ['ka-Geok', 0x10A0, 0x10A1],
'yue, ZHH' => ['yue', 0x9AA8, 0x4E09],
'yue-Hant-HK, ZHH' => ['yue-Hant-HK', 0x9AA8, 0x4E09],
'cmn-Hans, ZHS' => ['cmn-Hans', 0x9AA8, 0x4E00],
'lzh, ZHT' => ['lzh', 0x9AA8, 0x4E8C],
'zh-yue, ZHH' => ['zh-yue', 0x9AA8, 0x4E09],
'zh-lzh, ZHT' => ['zh-lzh', 0x9AA8, 0x4E8C],
'ga, IRT after IRI' => ['ga', 0x61, 0x69],
'nv, ATH after NAV' => ['nv', 0x61, 0x74],
'cmn-Hant-TW, ZHT' => ['cmn-Hant-TW', 0x9AA8, 0x4E8C],
'yue-Hans, ZHS' => ['yue-Hans', 0x9AA8, 0x4E00],
'ro, ROM not offered' => ['ro', 0x61, 0x61],
];
}

}
Loading
Loading