diff --git a/CLAUDE.md b/CLAUDE.md index fcb5ce2..8e08f66 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,7 +90,7 @@ These are the non-obvious constraints; the rest of the architecture is in `docs/ - **Time comes from `ClockService`.** Pure time utils take `now: Date` as a parameter and callers pass `clock.now()`. Reading `new Date()` inside a `computed()` freezes the value: the computed depends on no signal representing time, so a card shows "4 min ago" forever. - **CSP is on.** `src-tauri/tauri.conf.json` locks the WebView down; `ipc:` and `http://ipc.localhost` must stay in `connect-src` or `invoke()` is blocked. Anything remote (fonts, images, APIs) needs a deliberate widening — fonts are self-hosted via `@fontsource` for exactly this reason. - **The Rust model structs are a _contract_.** Don't change the shape of anything in `notes/model.rs`, `notes/view.rs` or `spaces/model.rs` without changing the DTOs on the front; their serde tests will fail if you do. -- **Search matching is Rust, not SQL.** SQLite's `LOWER()` only folds ASCII without ICU, so a `WHERE LOWER(title) LIKE …` would stop matching `Étape` against `étape`. Coarse filters (space, pin, lifecycle, language, tags) stay in SQL where they're indexed; text matching runs on the fetched rows via `to_lowercase()`. +- **Search matching is Rust, not SQL.** SQLite's `LOWER()` only folds ASCII without ICU, so a `WHERE LOWER(title) LIKE …` would stop matching `Étape` against `étape`. Coarse filters (space, pin, lifecycle, language, tags) stay in SQL where they're indexed; text matching runs on the fetched rows through `view::fold`, which lowercases **and** strips the accents — `etape` finds `Étape`, and the needle goes through the same fold, so it works the other way round too. Tag normalisation does not fold accents: two spellings there are two tags. - **Syntax highlighting is highlight.js, front-side, in exactly one module.** `notes/ui/code-viewer/highlighter.ts` imports grammars **one by one** (`highlight.js/lib/languages/…`), never the default bundle. It colours the whole block — that's what handles multi-line comments and strings — then re-splits the output with `splitHighlightedLines`, which reopens the tag stack across each newline. The `.hljs-*` theme lives in the **global** `src/styles/_code-theme.scss`: injected by `[innerHTML]`, it carries no `_ngcontent` attribute, so a component-scoped rule would never match. - **Tag normalisation lives in `notes::model::normalize_tags`, and only there.** Trim, strip leading `#`, drop blanks, collapse case-insensitive duplicates. `NoteDraft::into_note` and `NotePatch::apply` call it; `notes::store::replace_tags` receives tags already normalised and is pure SQL. It **re-reads** them after writing rather than sorting: `note_tags.tag` is `COLLATE NOCASE` (migration 2) and a read orders in that collation, which a byte-wise `sort()` does not reproduce — `Urgent` would come back before `auth` on write and after it on reload. - **A `computed` feeding a `resource` needs an `equal` comparator.** `resource` compares params by identity. `NotesQueryStore.queryParams` returns a fresh object literal and reads `clock.now()`: without `sameQueryParams`, every 30 s tick fired a full `query_notes` round trip, invisible behind the retained view. diff --git a/docs/architecture.md b/docs/architecture.md index 9902924..ec7f39d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1602,6 +1602,17 @@ installed or shipped alongside the executable. The database file lives in Tauri' matching is done **in Rust** (`notes::view`), because SQLite's `LOWER()` only folds ASCII without ICU, so `Étape` would not match `étape`. Grouping is `notes::view`, which touches no connection and is therefore testable without a database. +- **The search fold drops the accents as well as the case, on both sides.** `view::fold` + lowercases and strips the combining marks, so `etape` finds `Étape` and `Étape` finds + `etape` — the needle goes through the same function as every haystack. It decomposes one + character at a time (`unicode_normalization::char::decompose_canonical`) rather than + streaming the whole string through `nfd()`: on 800 notes of 13 kB of accented text, in + release, that is 15 ms against 76 ms, and against 27 ms for the `to_lowercase()` it + replaces — which folded no accent at all. Pure ASCII, which is what a snippet of code is, + never leaves the fast path. Only what a **canonical** decomposition separates is folded: + `ø` and `ß` are letters in their own right and stay. ⚠️ Tag normalisation + (`notes::model::normalize_tags`) deliberately does **not** fold accents — `Étape` and + `etape` are two tags, and merging them would lose one. - **Tag normalisation lives in `notes::model::normalize_tags`, and only there.** Trimming, stripping leading `#`, dropping blanks and collapsing case-insensitive duplicates (first spelling wins) all happen on write, so the front sends what the user typed. The returned diff --git a/e2e/specs/06-search-and-filters.e2e.ts b/e2e/specs/06-search-and-filters.e2e.ts index 631afc1..04d52df 100644 --- a/e2e/specs/06-search-and-filters.e2e.ts +++ b/e2e/specs/06-search-and-filters.e2e.ts @@ -60,6 +60,16 @@ describe('Search, filters and facets', () => { expect(await canvas.titles()).toEqual(['Étape de migration']); }); + it('folds the accents too, on both sides of the comparison', async () => { + // Nobody reaches for the accent key to search, and the corpus is written with them. + await canvas.search('etape'); + expect(await canvas.titles()).toEqual(['Étape de migration']); + + // Symmetric, because the needle goes through the same fold as the haystack. + await canvas.search('Dôcker'); + expect(await canvas.titles()).toEqual(['Docker compose']); + }); + it('collapses to a single flat results section while searching', async () => { // Searched here rather than inherited from the test above: an `it` that depends on // what the previous one left cannot be run, reordered or bailed on alone. diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index 0e780f2..1706ccd 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -934,6 +934,7 @@ dependencies = [ "tauri-specta", "thiserror 2.0.20", "toml 1.1.3+spec-1.1.0", + "unicode-normalization", "uuid", ] @@ -5385,6 +5386,15 @@ version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" +[[package]] +name = "unicode-normalization" +version = "0.1.25" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fd4f6878c9cb28d874b009da9e8d183b5abc80117c40bbd187a1fde336be6e8" +dependencies = [ + "tinyvec", +] + [[package]] name = "unicode-segmentation" version = "1.13.3" diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index e14a472..4524843 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -50,6 +50,10 @@ tauri-plugin-clipboard-manager = "2" tauri-plugin-log = "2" log = "0.4" thiserror = "2.0.20" +# Search folds accents, which means decomposing before dropping the marks: `etape` +# has to find `Étape` on a French corpus. Hand-rolling a Latin table instead would +# be a list that goes stale the first time someone writes Polish or Vietnamese. +unicode-normalization = "0.1.25" # Pulled in only by the `e2e` feature, and not optional in practice: the runner # waits on `window.wdioTauri` before it will open a session, so a binary without # the plugin hangs the suite rather than failing it. Its Rust half and the npm half diff --git a/src-tauri/src/notes/view.rs b/src-tauri/src/notes/view.rs index 35ece3c..a398dbe 100644 --- a/src-tauri/src/notes/view.rs +++ b/src-tauri/src/notes/view.rs @@ -3,6 +3,7 @@ use std::collections::{BTreeMap, HashMap}; use chrono::{DateTime, Datelike, FixedOffset, TimeDelta, Utc}; use serde::{Deserialize, Serialize}; use specta::Type; +use unicode_normalization::char::{decompose_canonical, is_combining_mark}; use super::language::Language; use super::model::{self, DisplayNote, Note}; @@ -104,7 +105,7 @@ pub fn apply_global_defaults(view: &mut NotesView, globals: &BTreeMap, facets: Facets, request: &NotesQuery) -> NotesView { - let needle = request.search.trim().to_lowercase(); + let needle = fold(request.search.trim()); if !needle.is_empty() { notes.retain(|note| matches_search(note, &needle)); } @@ -136,7 +137,7 @@ pub fn build(mut notes: Vec, facets: Facets, request: &NotesQuery) -> Note } } -/// `needle` is expected **already folded to lowercase and trimmed**. +/// `needle` is expected to have been through [`fold`] and trimmed. /// /// ⚠️ Folded in Rust and not in SQL: without ICU, SQLite's `LOWER()` only handles /// ASCII, so `Étape` would not match `étape`. The items count as much as the @@ -151,12 +152,52 @@ fn matches_search(note: &Note, needle: &str) -> bool { .any(|item| contains_folded(&item.text, needle)) } +/// Lowercase **and** accent-free, so `etape` finds `Étape`. Both sides go through +/// here, which makes the match symmetric: an accented needle finds unaccented text +/// too, since a search field is not where anyone wants to be precise about it. +/// +/// Decomposing to NFD and dropping the combining marks handles every script rather +/// than the Latin letters someone thought to list. Only what a canonical +/// decomposition separates is folded: `ø` and `ß` are letters of their own and stay. +/// +/// ⚠️ Decomposed one character at a time, and not through the `nfd()` iterator over +/// the whole string: on 800 notes of 13 kB of accented text, release, the streaming +/// version cost 76 ms against 15 ms here for the same answer. Its lookahead buffering +/// earns nothing when every mark is dropped anyway — canonical order cannot matter to +/// a fold that keeps none of it. +/// +/// The ASCII branches are the common ones, not micro-optimisations: a snippet of code +/// is ASCII from end to end, and a French sentence is ASCII between its accents. That +/// is also what pays for the accents: the same corpus took 27 ms through +/// `to_lowercase()`, which folded no accent at all. +fn fold(text: &str) -> String { + if text.is_ascii() { + return text.to_ascii_lowercase(); + } + + let mut folded = String::with_capacity(text.len()); + + for character in text.chars() { + if character.is_ascii() { + folded.push(character.to_ascii_lowercase()); + } else { + decompose_canonical(character, |part| { + if !is_combining_mark(part) { + folded.extend(part.to_lowercase()); + } + }); + } + } + + folded +} + /// ⚠️ Do **not** hand-roll a fold-as-you-compare scan to save the copy: measured /// on 800 notes of 13 kB, a needle matching nothing took 11.0 ms that way against /// 6.3 ms here. `str::contains` runs Two-Way (O(n+m)); a window scan is O(n·m), /// and searching is precisely the case where most notes do not match. fn contains_folded(haystack: &str, needle: &str) -> bool { - haystack.to_lowercase().contains(needle) + fold(haystack).contains(needle) } const A_WEEK: TimeDelta = TimeDelta::days(7); @@ -320,7 +361,7 @@ mod tests { notes, Facets::default(), &NotesQuery { - search: " DÉPLOI ".to_string(), + search: " DEPLOI ".to_string(), ..request() }, ); @@ -419,10 +460,10 @@ mod tests { ..sample() }; - assert!(matches_search(¬e, "déploi")); - assert!(matches_search(¬e, "kubectl")); - assert!(matches_search(¬e, "ops")); - assert!(!matches_search(¬e, "terraform")); + assert!(matches_search(¬e, &fold("déploi"))); + assert!(matches_search(¬e, &fold("kubectl"))); + assert!(matches_search(¬e, &fold("ops"))); + assert!(!matches_search(¬e, &fold("terraform"))); } #[test] @@ -432,7 +473,52 @@ mod tests { ..sample() }; - assert!(matches_search(¬e, "étape")); + assert!(matches_search(¬e, &fold("étape"))); + assert!(matches_search(¬e, &fold("ÉTAPE"))); + } + + /// The point of the whole fold: nobody reaches for the accent key to search. + #[test] + fn an_unaccented_needle_finds_accented_text() { + let note = Note { + title: "Étape de migration".to_string(), + content: "Prévenir l'équipe".to_string(), + tags: vec!["déploiement".to_string()], + ..sample() + }; + + assert!(matches_search(¬e, &fold("etape"))); + assert!(matches_search(¬e, &fold("equipe"))); + assert!(matches_search(¬e, &fold("deploiement"))); + } + + /// Both sides go through `fold`, so it works the other way round too. + #[test] + fn an_accented_needle_finds_unaccented_text() { + let note = Note { + title: "Etape de migration".to_string(), + ..sample() + }; + + assert!(matches_search(¬e, &fold("Étape"))); + } + + /// Not everything an eye reads as an accent is one: only what a canonical + /// decomposition separates is folded away. + #[test] + fn a_letter_of_its_own_is_not_folded_into_another() { + let note = Note { + title: "Størrelse".to_string(), + ..sample() + }; + + assert!(matches_search(¬e, &fold("størrelse"))); + assert!(!matches_search(¬e, &fold("storrelse"))); + } + + #[test] + fn folding_leaves_an_ascii_needle_alone() { + assert_eq!(fold("Kubectl APPLY"), "kubectl apply"); } #[test] @@ -454,9 +540,9 @@ mod tests { ..sample() }; - assert!(matches_search(¬e, "migration")); - assert!(matches_search(¬e, "équipe")); - assert!(!matches_search(¬e, "terraform")); + assert!(matches_search(¬e, &fold("migration"))); + assert!(matches_search(¬e, &fold("equipe"))); + assert!(!matches_search(¬e, &fold("terraform"))); } }