Skip to content

Fold the accents in search, not just the case - #94

Merged
vmillet-dev merged 1 commit into
mainfrom
fix/search-folds-accents
Sep 13, 2026
Merged

vmillet-dev merged 1 commit into
mainfrom
fix/search-folds-accents

Conversation

@vmillet-dev

@vmillet-dev vmillet-dev commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Closes #16.

contains_folded lowercased and nothing more, so a needle typed without accents never matched accented text: etape did not find Étape 1. On a French interface that is a daily miss.

The decision the ticket asked for: the fold is symmetric

Both sides go through the same function. etape finds Étape, and Étape finds etape. A search field is not where anyone wants to be precise about diacritics, and an asymmetric fold would mean the result depends on which side of the comparison the accent happened to land.

How it folds

NFD decomposition, then the combining marks are dropped, then lowercase. That handles every script rather than the Latin letters someone thought to list — which is also why the ticket's alternative, a hand-written table, was not taken: it goes stale the first time someone writes Polish or Vietnamese.

Only what a canonical decomposition separates is folded. ø and ß are letters in their own right, not accented letters, and stay as they are — there is a test saying so, because it is a judgement and not an oversight.

⚠️ Tag normalisation (notes::model::normalize_tags) deliberately does not fold accents. Étape and etape are two tags; merging them would lose one.

The measurement the ticket asked for

The existing comment is right that the fold has to happen once per haystack with str::contains kept, and the naive way hands the measurement straight back. Measured on the ticket's shape — 800 notes of 13 kB, a needle matching nothing, release build, best of seven:

corpus to_lowercase() (before) nfd() over the whole string per-character (shipped)
ASCII body 0.5 ms 0.6 ms
accented body 26.5 ms 75.8 ms 15.1 ms

Two things came out of that:

  • The streaming nfd() iterator is the expensive way. Its lookahead buffering earns nothing here — canonical ordering cannot matter to a fold that keeps none of the marks. unicode_normalization::char::decompose_canonical on one character at a time gives the same answer for a fifth of the cost.
  • ASCII must not pay for this. A snippet of code has no accent in it, and a French sentence is ASCII between its accents, so both get a branch of their own. That is what makes the accented case come out faster than before: 15.1 ms against the 26.5 ms of a to_lowercase() that folded no accent at all. ASCII costs 0.1 ms more, for the is_ascii() scan.

The benchmark itself was a throwaway — the repository records measurements in the comment next to the code, as the existing one does, rather than carrying a bench harness.

Dependency

One new crate, unicode-normalization (MIT OR Apache-2.0, in deny.toml's allow list). It brought nothing with it: tinyvec was already in the tree. bindings.ts is unchanged — nothing crossing the bridge moved.

Tests

Rust, in notes::view: an unaccented needle finding accented title, content and tags; an accented needle finding unaccented text; ø not folding into o; and the existing search tests now pass their needle through fold, which is what the contract on matches_search actually asks for.

e2e, in 06-search-and-filters: etape finds Étape de migration and Dôcker finds Docker compose, through the real bridge.

Checked

  • cargo test — 161 + 34 + 83 passing.
  • cargo clippy --all-targets -- -D warnings and cargo fmt --check — clean.
  • npm test — 916 passing. npm run lint — clean.
  • e2e 01 and 06 — green, including the two new assertions.

docs/architecture.md and CLAUDE.md updated with the fold and the tag-normalisation exception

`contains_folded` lowercased and nothing more, so `etape` never found
`Étape` — a daily miss on a French corpus. `fold` now lowercases and
drops the combining marks, and the needle goes through it too, so an
accented needle finds unaccented text just as well.

It decomposes one character at a time rather than streaming the string
through `nfd()`, and leaves pure ASCII alone: 15 ms on 800 notes of 13 kB
of accented text against 76 ms streaming, and against 27 ms for the
`to_lowercase()` it replaces, which folded no accent at all.
@vmillet-dev
vmillet-dev merged commit abeb974 into main Sep 13, 2026
10 of 11 checks passed
@vmillet-dev
vmillet-dev deleted the fix/search-folds-accents branch September 13, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Search does not fold accents: etape does not find Étape

1 participant