From 42349235a863b5c518c05f8c7c2fe2829a802ac9 Mon Sep 17 00:00:00 2001 From: Onur Cinar Date: Sun, 6 Sep 2026 16:02:07 +0000 Subject: [PATCH] Add published article on invisible characters and NFKC normalization Archives the dev.to post covering strip-invisible (Trojan Source-style bidi/zero-width spoofing) and the opt-in nfkc module, matching the existing articles/ convention of committing the piece after it goes live. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_019gjhjkaubxHpwNQ1ve23vy --- ...-unicode-normalization-in-go-validation.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 articles/invisible-characters-and-unicode-normalization-in-go-validation.md diff --git a/articles/invisible-characters-and-unicode-normalization-in-go-validation.md b/articles/invisible-characters-and-unicode-normalization-in-go-validation.md new file mode 100644 index 0000000..59c5f5d --- /dev/null +++ b/articles/invisible-characters-and-unicode-normalization-in-go-validation.md @@ -0,0 +1,94 @@ +--- +title: The Go Validator Blind Spot: Invisible Characters and Unicode Lookalikes +published: true +description: A username that reads "admin" on every screen you check it on can be storing something else entirely -- and it'll sail past a reserved-name check that's doing exactly what it was told. Most Go validation stacks have no way to catch it. Here's why, and two small normalizers that close the gap. +tags: go, golang, security, unicode +canonical_url: https://dev.to/onurcinar/the-go-validator-blind-spot-invisible-characters-and-unicode-lookalikes-38j5 +--- + +Here's a username: `‮nimda‬`. Depending on how your browser renders bidirectional text, that might already display as `admin` — which is the entire point. Read it in a user list, a support ticket, or an audit log, and every person who looks at it will tell you the account is named `admin`. Now compare it to the literal string, the way a reserved-name check normally works: + +```go +spoofed := "‮nimda‬" // renders as "admin" wherever bidi text is displayed + +fmt.Println(spoofed == "admin") // false +fmt.Println(len([]rune(spoofed))) // 7, not 5 +``` + +It isn't `"admin"`. It's the five letters `n`, `i`, `m`, `d`, `a` — in that literal order — wrapped in two invisible bidirectional control characters, U+202E (right-to-left override) and U+202C (pop directional formatting). Unicode reserves those for rendering Arabic and Hebrew correctly. Wrapped around plain Latin letters instead, they just flip the visual order: `nimda`, laid out right-to-left, reads `admin`. + +A reserved-username check that rejects the exact string `"admin"` lets this straight through — correctly, by its own logic, since the string genuinely isn't `"admin"`. Nothing here is a bug in `checker`, or in any other Go validation library, or even in Unicode. The check did exactly what it was told. The problem is one level up: it was never asked whether the string secretly contains characters that don't correspond to anything the comparison sees but everyone looking at a screen does. + +## This is not a new attack + +The trick behind that username has a name: in November 2021, researchers at Cambridge published [Trojan Source (CVE-2021-42574)](https://trojansource.codes/), showing that the same bidirectional control characters can be slipped into source code to make it *compile* one way and *display* another way to a human reviewer. The characters are identical; only the target changes. Anything a person reads back instead of a compiler — a username, a chat message, a support ticket, a PR title — is just as susceptible. + +A narrower, older trick uses zero-width characters instead of bidi ones: insert a zero-width space or zero-width joiner into a blocklisted word and a naive filter no longer matches it, even though the word displays unchanged. It's a standard way spam and moderation filters get evaded, and it works against a validation struct tag exactly as well as it works against a regex. + +## The other half of the problem: it doesn't need to be invisible at all + +Zero-width characters are the sharp case, but plain Unicode gives you the same problem with characters you *can* see. `"ALICE"`, `"ＡＬＩＣＥ"` (fullwidth Latin letters), and a version using ligatures or Roman numerals in place of ordinary letters are three different byte sequences that a human reading them would call the same word. A uniqueness check, a keyword filter, or a duplicate-handle guard that only compares strings byte-for-byte treats all three as unrelated. + +This isn't hypothetical, either — fullwidth-character substitution is a known way to slip a blocked command past a keyword-based filter (`ｒｍ　－ｒｆ　／` reads as `rm -rf /` to anyone who looks at it, and normalizes right back to it), and several platforms have had to retrofit Unicode normalization after discovering it let people register visually-identical duplicate usernames. + +None of this is because validation is careless. `required`, `min-len`, an exact-match reserved-name rule — every one of them is a byte-level check, and byte-level checks were never asked a different question: does this string contain characters that don't correspond to anything the person reading it actually typed? That's a normalization problem, not a validation one. It needs its own step in the pipeline, before anything else looks at the value. + +## Two checkers, two different problems + +`checker` added two normalizers for this, deliberately kept separate because they solve different problems and have different costs. + +**`strip-invisible`** (core module, no dependency) removes zero-width space, zero-width non-joiner, zero-width joiner, word joiner, the zero-width no-break space (BOM), and the bidirectional embedding/override/isolate controls behind Trojan Source: + +```go +type Handle struct { + Name string `checkers:"trim strip-invisible required min-len:3"` +} + +h := &Handle{Name: "adm‌in"} +checker.CheckStruct(h) +// h.Name is now "admin" -- five runes, matches the literal string. + +unmasked, _ := checker.StripInvisible("‮nimda‬") +fmt.Println(unmasked) // nimda +fmt.Println(unmasked == "admin") // false -- and now it doesn't even display like it +``` + +Run the username from the opening through it and the disguise falls apart: `strip-invisible` doesn't know or care what the override characters were trying to accomplish, it just removes them — leaving the honest `nimda`, which no longer reads as `admin` in any renderer. + +**`nfkc`** (an opt-in module — it needs `golang.org/x/text/unicode/norm`, so it's kept out of the dependency-free core) applies [Unicode Normalization Form KC](https://unicode.org/reports/tr15/): it folds compatibility characters — fullwidth forms, ligatures, and similar stylistic variants — into their canonical equivalents: + +```go +import _ "github.com/cinar/checker/v2/nfkc" + +type Handle struct { + Name string `checkers:"trim nfkc required min-len:3"` +} + +a := &Handle{Name: "ＡＬＩＣＥ"} +checker.CheckStruct(a) +// a.Name is now "ALICE" + +waf, _ := nfkc.Normalize("ｒｍ　－ｒｆ　／") +// waf is "rm -rf /" +``` + +Both are one word in a `checkers` tag. Neither needs a second pass over your validation logic — they sit in the same pipeline as `trim`, `lower`, and `required`, because normalizers and checkers are the same function type in this library and can be freely mixed. + +## What this does *not* fix + +Be precise about what NFKC actually covers, because it's narrower than "Unicode spoofing" in general. NFKC only folds characters that are *compatibility-equivalent* under the Unicode standard — the same letter, rendered differently. It does nothing for **homoglyphs**: characters that are visually similar but come from a different script entirely, like a Cyrillic `а` (U+0430) standing in for a Latin `a` (U+0061). Those two are not NFKC-equivalent, so `paypal.com` spelled with a Cyrillic `а` normalizes right back to itself — still spoofed. + +Catching that class of lookalike needs a *confusables* check — [Unicode UTS #39](https://unicode.org/reports/tr39/)'s skeleton algorithm, which maps visually similar characters across scripts to a shared representative form before comparing. It's a genuinely different, heavier piece of machinery than either normalizer here, and `checker` doesn't currently provide it. If cross-script impersonation is part of your threat model — a login handle, a domain-like identifier — `strip-invisible` plus `nfkc` narrows the problem but doesn't close it. + +## Where to apply this, and where not to + +Both normalizers are opinionated about *removing or rewriting* characters, which means they're right for the fields where an invisible or compatibility character is never legitimate — a username, a handle, a search keyword, an API key — and wrong for general free-text content. Zero-width joiner is load-bearing in emoji sequences; zero-width non-joiner has real uses in Persian and other scripts. Running `strip-invisible` over a chat message body or a comment field will quietly corrupt text some of your users actually rely on. Point these at identifiers, not prose. + +```go +type Registration struct { + Handle string `checkers:"trim strip-invisible nfkc required min-len:3 alphanumeric"` + Bio string `checkers:"trim max-len:500"` // free text -- neither normalizer applies here +} +``` + +`strip-invisible` ships in [Checker](https://github.com/cinar/checker)'s core module; `nfkc` is a separate, opt-in module at [`github.com/cinar/checker/v2/nfkc`](https://github.com/cinar/checker/tree/main/nfkc), so the dependency it needs never touches a codebase that doesn't ask for it.