Fold the accents in search, not just the case - #94
Merged
Merged
Conversation
`contains_folded` lowercased and nothing more, so `etape` never found `Étape` — a daily miss on a French corpus. `fold` now lowercases and drops the combining marks, and the needle goes through it too, so an accented needle finds unaccented text just as well. It decomposes one character at a time rather than streaming the string through `nfd()`, and leaves pure ASCII alone: 15 ms on 800 notes of 13 kB of accented text against 76 ms streaming, and against 27 ms for the `to_lowercase()` it replaces, which folded no accent at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #16.
contains_foldedlowercased and nothing more, so a needle typed without accents never matched accented text:etapedid not findÉtape 1. On a French interface that is a daily miss.The decision the ticket asked for: the fold is symmetric
Both sides go through the same function.
etapefindsÉtape, andÉtapefindsetape. A search field is not where anyone wants to be precise about diacritics, and an asymmetric fold would mean the result depends on which side of the comparison the accent happened to land.How it folds
NFD decomposition, then the combining marks are dropped, then lowercase. That handles every script rather than the Latin letters someone thought to list — which is also why the ticket's alternative, a hand-written table, was not taken: it goes stale the first time someone writes Polish or Vietnamese.
Only what a canonical decomposition separates is folded.
øandßare letters in their own right, not accented letters, and stay as they are — there is a test saying so, because it is a judgement and not an oversight.notes::model::normalize_tags) deliberately does not fold accents.Étapeandetapeare two tags; merging them would lose one.The measurement the ticket asked for
The existing comment is right that the fold has to happen once per haystack with
str::containskept, and the naive way hands the measurement straight back. Measured on the ticket's shape — 800 notes of 13 kB, a needle matching nothing, release build, best of seven:to_lowercase()(before)nfd()over the whole stringTwo things came out of that:
nfd()iterator is the expensive way. Its lookahead buffering earns nothing here — canonical ordering cannot matter to a fold that keeps none of the marks.unicode_normalization::char::decompose_canonicalon one character at a time gives the same answer for a fifth of the cost.to_lowercase()that folded no accent at all. ASCII costs 0.1 ms more, for theis_ascii()scan.The benchmark itself was a throwaway — the repository records measurements in the comment next to the code, as the existing one does, rather than carrying a bench harness.
Dependency
One new crate,
unicode-normalization(MIT OR Apache-2.0, indeny.toml's allow list). It brought nothing with it:tinyvecwas already in the tree.bindings.tsis unchanged — nothing crossing the bridge moved.Tests
Rust, in
notes::view: an unaccented needle finding accented title, content and tags; an accented needle finding unaccented text;ønot folding intoo; and the existing search tests now pass their needle throughfold, which is what the contract onmatches_searchactually asks for.e2e, in
06-search-and-filters:etapefindsÉtape de migrationandDôckerfindsDocker compose, through the real bridge.Checked
cargo test— 161 + 34 + 83 passing.cargo clippy --all-targets -- -D warningsandcargo fmt --check— clean.npm test— 916 passing.npm run lint— clean.01and06— green, including the two new assertions.docs/architecture.mdandCLAUDE.mdupdated with the fold and the tag-normalisation exception