diff --git a/docs/features/library.md b/docs/features/library.md index 1aab5e47..cba9b32d 100644 --- a/docs/features/library.md +++ b/docs/features/library.md @@ -190,6 +190,7 @@ Which columns the track table shows, in what order and how wide, per profile in Five things that are easy to get wrong here, and are not: +- **Names sort under `FOLD`, not `NOCASE`.** SQLite's `NOCASE` folds ASCII only, and the scanner's `canonical_name` / `canonical_title` keep their accents, so "Émilie" sorted after "Zazie" while the A-Z rail filed her under E. [`collation.rs`](../../src-tauri/crates/core/src/repository/sqlite/collation.rs) defines a `FOLD` collation that sets case and accents aside (every decomposable accent through NFD, plus the letters NFD leaves whole, so a mirrored row's `normalize_name` key sorts next to a local one that kept its accents; only then are ties broken on the lowercased and the original strings, so two distinct keys never compare equal), then breaks ties on case and bytes so the order stays total. The profile pool registers it on every connection (`collation::register` in [`profile_db.rs`](../../src-tauri/crates/app/src/db/profile_db.rs)); a query naming it on a connection opened any other way fails to prepare, so a test pool that runs a listing registers it too. Paths, codecs, keys and the folder browser's grouping keep `NOCASE`. - **Sorting happens in SQL, over a whitelist.** `browse::track_sort_expr` maps a column id to an expression; anything unmatched falls back to the default clause, which is the only reason the caller may hand the result to `AssertSqlSafe`. Nullable columns put their NULLs last **in both directions** — an untagged track is not a track with the smallest value — and every sort carries a tie-break so two identical queries cannot return two different orders. The previous shape spelled out two `match` arms per column and had quietly stopped at seven of them. - **The resize handle lives outside the sort button.** Inside it, a press to widen the column fires the sort on release, so every resize reorders the list. - **"Fit to content" measures through a canvas, not the DOM.** The rows are virtualised, so the ones outside the viewport have no computed layout and a DOM measurement would size the column to whatever happened to be on screen. The **header label is measured too**, in the header's own (uppercase, bold, letter-spaced) font: a column fitted to short content otherwise shows a truncated title, which reads as the fit having failed. diff --git a/docs/rfcs/RFC-005-remote-source-and-sync-v2.md b/docs/rfcs/RFC-005-remote-source-and-sync-v2.md index 2158a41d..64d5ede3 100644 --- a/docs/rfcs/RFC-005-remote-source-and-sync-v2.md +++ b/docs/rfcs/RFC-005-remote-source-and-sync-v2.md @@ -810,16 +810,26 @@ RFC — now written, as Two things the query has to get right, and both are about the halves being comparable rather than merely concatenated: -- **The sort keys are normalised on both sides.** The local half sorts on - `album.canonical_title` / `artist.canonical_name` — forms produced by - `normalize_name`, which lowercases, folds diacritics and drops punctuation. - SQLite cannot reproduce any of that (`COLLATE NOCASE` is ASCII-only), so - sorting the remote half on its raw display name puts "Björk" and "bjork" in - two different places and splits one artist in half down the middle of the - list. `remote_album.sort_title` / `sort_artist` therefore carry the same - normalised forms, computed by the mirror with the same function. A row - mirrored before those columns existed falls back to its display title, and - one walk fills it in. +- **The sort keys are comparable on both sides.** The two halves do not + spell them the same way: + - the local half sorts on `album.canonical_title` / + `artist.canonical_name`, the scanner's `canonical_name`: lowercased, + punctuation dropped, **accents kept**; + - the remote half sorts on `remote_album.sort_title` / `sort_artist`, + which the mirror computes with `normalize_name`: lowercased, punctuation + dropped, **accents folded**, `&` spelled "and". + + Sorting the remote half on its raw display name instead put "Björk" and + "bjork" in two different places and split one artist in half down the + middle of the list. The unified query projects each half's key under one + alias and orders it with `COLLATE FOLD` (see + [library.md](../features/library.md)), which sets case and accents aside + first, so a local key that kept its accents sorts next to a remote one + that folded them (case and the original characters only break the tie + between them). `COLLATE NOCASE`, which the listings used + before, is ASCII-only and could not. A row mirrored before the sort + columns existed has them `NULL`, and `COALESCE` supplies its display title + and artist until a mirror walk fills them in. - **A local library filter excludes the remote half.** The picker chooses among _local_ libraries, and a server album belongs to none of them; leaving the remote rows visible while the user has narrowed to one library reads as the diff --git a/packaging/flatpak/generated/cargo-sources.json b/packaging/flatpak/generated/cargo-sources.json index 391c213e..64e883df 100644 --- a/packaging/flatpak/generated/cargo-sources.json +++ b/packaging/flatpak/generated/cargo-sources.json @@ -8738,14 +8738,14 @@ { "type": "archive", "archive-type": "tar-gzip", - "url": "https://static.crates.io/crates/wasapi/wasapi-0.24.0.crate", - "sha256": "ee144c215d8d31e94c1a599fbaf540ca82a7017e4f1786816b179f334fac751d", - "dest": "cargo/vendor/wasapi-0.24.0" + "url": "https://static.crates.io/crates/wasapi/wasapi-0.25.0.crate", + "sha256": "5225a7c681bdaf6a63aa0ffc287adba7fd5a04b592a2730643869ce45641da99", + "dest": "cargo/vendor/wasapi-0.25.0" }, { "type": "inline", - "contents": "{\"package\": \"ee144c215d8d31e94c1a599fbaf540ca82a7017e4f1786816b179f334fac751d\", \"files\": {}}", - "dest": "cargo/vendor/wasapi-0.24.0", + "contents": "{\"package\": \"5225a7c681bdaf6a63aa0ffc287adba7fd5a04b592a2730643869ce45641da99\", \"files\": {}}", + "dest": "cargo/vendor/wasapi-0.25.0", "dest-filename": ".cargo-checksum.json" }, { diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index e24195c4..fd8f00d7 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -7825,6 +7825,7 @@ dependencies = [ "tokio", "toml 1.1.6+spec-1.1.0", "tracing", + "unicode-normalization", "usvg", "uuid 1.27.0", "wasmtime", diff --git a/src-tauri/crates/app/src/commands/browse.rs b/src-tauri/crates/app/src/commands/browse.rs index 1d18bbae..aebff171 100644 --- a/src-tauri/crates/app/src/commands/browse.rs +++ b/src-tauri/crates/app/src/commands/browse.rs @@ -240,15 +240,15 @@ fn album_order_clause(order_by: Option<&str>, direction: Option<&str>) -> &'stat } }; match (order_by, dir) { - (Some("title"), "ASC") => "ORDER BY al.canonical_title COLLATE NOCASE ASC", - (Some("title"), "DESC") => "ORDER BY al.canonical_title COLLATE NOCASE DESC", - (Some("artist"), "ASC") => "ORDER BY ar.canonical_name COLLATE NOCASE ASC, al.canonical_title COLLATE NOCASE", - (Some("artist"), "DESC") => "ORDER BY ar.canonical_name COLLATE NOCASE DESC, al.canonical_title COLLATE NOCASE", - (Some("year"), "ASC") => "ORDER BY al.year ASC, al.canonical_title COLLATE NOCASE", - (Some("year"), "DESC") => "ORDER BY al.year DESC, al.canonical_title COLLATE NOCASE", + (Some("title"), "ASC") => "ORDER BY al.canonical_title COLLATE FOLD ASC", + (Some("title"), "DESC") => "ORDER BY al.canonical_title COLLATE FOLD DESC", + (Some("artist"), "ASC") => "ORDER BY ar.canonical_name COLLATE FOLD ASC, al.canonical_title COLLATE FOLD", + (Some("artist"), "DESC") => "ORDER BY ar.canonical_name COLLATE FOLD DESC, al.canonical_title COLLATE FOLD", + (Some("year"), "ASC") => "ORDER BY al.year ASC, al.canonical_title COLLATE FOLD", + (Some("year"), "DESC") => "ORDER BY al.year DESC, al.canonical_title COLLATE FOLD", (Some("added_at"), "ASC") => "ORDER BY MIN(t.added_at) ASC", (Some("added_at"), "DESC") => "ORDER BY MIN(t.added_at) DESC", - _ => "ORDER BY ar.canonical_name COLLATE NOCASE,\n al.canonical_title COLLATE NOCASE", + _ => "ORDER BY ar.canonical_name COLLATE FOLD,\n al.canonical_title COLLATE FOLD", } } @@ -267,21 +267,17 @@ fn artist_order_clause(order_by: Option<&str>, direction: Option<&str>) -> &'sta } }; match (order_by, dir) { - (Some("name"), "ASC") => "ORDER BY ar.canonical_name COLLATE NOCASE ASC", - (Some("name"), "DESC") => "ORDER BY ar.canonical_name COLLATE NOCASE DESC", - (Some("albums_count"), "ASC") => { - "ORDER BY album_count ASC, ar.canonical_name COLLATE NOCASE" - } + (Some("name"), "ASC") => "ORDER BY ar.canonical_name COLLATE FOLD ASC", + (Some("name"), "DESC") => "ORDER BY ar.canonical_name COLLATE FOLD DESC", + (Some("albums_count"), "ASC") => "ORDER BY album_count ASC, ar.canonical_name COLLATE FOLD", (Some("albums_count"), "DESC") => { - "ORDER BY album_count DESC, ar.canonical_name COLLATE NOCASE" - } - (Some("tracks_count"), "ASC") => { - "ORDER BY track_count ASC, ar.canonical_name COLLATE NOCASE" + "ORDER BY album_count DESC, ar.canonical_name COLLATE FOLD" } + (Some("tracks_count"), "ASC") => "ORDER BY track_count ASC, ar.canonical_name COLLATE FOLD", (Some("tracks_count"), "DESC") => { - "ORDER BY track_count DESC, ar.canonical_name COLLATE NOCASE" + "ORDER BY track_count DESC, ar.canonical_name COLLATE FOLD" } - _ => "ORDER BY ar.canonical_name COLLATE NOCASE", + _ => "ORDER BY ar.canonical_name COLLATE FOLD", } } @@ -455,19 +451,17 @@ fn library_album_order_clause(order_by: Option<&str>, direction: Option<&str>) - } }; match (order_by, dir) { - (Some("title"), "ASC") => "ORDER BY sort_title COLLATE NOCASE ASC", - (Some("title"), "DESC") => "ORDER BY sort_title COLLATE NOCASE DESC", - (Some("artist"), "ASC") => { - "ORDER BY sort_artist COLLATE NOCASE ASC, sort_title COLLATE NOCASE" - } + (Some("title"), "ASC") => "ORDER BY sort_title COLLATE FOLD ASC", + (Some("title"), "DESC") => "ORDER BY sort_title COLLATE FOLD DESC", + (Some("artist"), "ASC") => "ORDER BY sort_artist COLLATE FOLD ASC, sort_title COLLATE FOLD", (Some("artist"), "DESC") => { - "ORDER BY sort_artist COLLATE NOCASE DESC, sort_title COLLATE NOCASE" + "ORDER BY sort_artist COLLATE FOLD DESC, sort_title COLLATE FOLD" } - (Some("year"), "ASC") => "ORDER BY year ASC, sort_title COLLATE NOCASE", - (Some("year"), "DESC") => "ORDER BY year DESC, sort_title COLLATE NOCASE", + (Some("year"), "ASC") => "ORDER BY year ASC, sort_title COLLATE FOLD", + (Some("year"), "DESC") => "ORDER BY year DESC, sort_title COLLATE FOLD", (Some("added_at"), "ASC") => "ORDER BY added_at ASC", (Some("added_at"), "DESC") => "ORDER BY added_at DESC", - _ => "ORDER BY sort_artist COLLATE NOCASE, sort_title COLLATE NOCASE", + _ => "ORDER BY sort_artist COLLATE FOLD, sort_title COLLATE FOLD", } } @@ -567,7 +561,8 @@ pub(crate) struct LibraryTrackRawRow { /// anything that is not matched here must produce the default clause /// rather than reaching a query. /// -/// Text columns sort case-insensitively; numeric ones do not, and every +/// Text columns sort case-insensitively (names under `FOLD`, which sets +/// accents aside too, so "Émilie" sits among the E); numeric ones do not, and every /// nullable one puts its NULLs last in both directions — an untagged /// track is not a track with the smallest value, and burying a run of /// blanks at the top of an ascending sort is the fastest way to make a @@ -575,9 +570,9 @@ pub(crate) struct LibraryTrackRawRow { fn track_sort_expr(key: &str) -> Option<(&'static str, bool)> { // (expression, nulls_last) Some(match key { - "title" => ("title COLLATE NOCASE", false), - "artist" => ("sort_artist COLLATE NOCASE", true), - "album" => ("sort_album COLLATE NOCASE", true), + "title" => ("title COLLATE FOLD", false), + "artist" => ("sort_artist COLLATE FOLD", true), + "album" => ("sort_album COLLATE FOLD", true), "duration_ms" => ("duration_ms", false), "year" => ("year", true), "added_at" => ("added_at", false), @@ -643,11 +638,11 @@ pub(crate) fn library_track_order_clause( // it: this is the order most lists are rendered in, so it is the one // where a pair of rows swapping places between two refreshes would // be seen most often. - const DEFAULT: &str = "ORDER BY sort_artist COLLATE NOCASE, - sort_album COLLATE NOCASE, + const DEFAULT: &str = "ORDER BY sort_artist COLLATE FOLD, + sort_album COLLATE FOLD, disc_number, track_number, - title COLLATE NOCASE, + title COLLATE FOLD, source, id"; @@ -674,9 +669,9 @@ pub(crate) fn library_track_order_clause( // album titles are not unique across artists (`Greatest Hits`, // `Live`, an untitled rip), and without it two such albums // interleave track 1 against track 1 all the way down. - Some("album") => clause.push_str(", sort_artist COLLATE NOCASE, disc_number, track_number"), + Some("album") => clause.push_str(", sort_artist COLLATE FOLD, disc_number, track_number"), Some("title") => {} - _ => clause.push_str(", title COLLATE NOCASE"), + _ => clause.push_str(", title COLLATE FOLD"), } // ...and then a **total** order. Two tracks of the same album with // the same disc and track number, or two files with the same title, @@ -1288,13 +1283,13 @@ fn library_artist_order_clause(order_by: Option<&str>, direction: Option<&str>) } }; match (order_by, dir) { - (Some("name"), "ASC") => "ORDER BY sort_name COLLATE NOCASE ASC", - (Some("name"), "DESC") => "ORDER BY sort_name COLLATE NOCASE DESC", - (Some("albums_count"), "ASC") => "ORDER BY album_count ASC, sort_name COLLATE NOCASE", - (Some("albums_count"), "DESC") => "ORDER BY album_count DESC, sort_name COLLATE NOCASE", - (Some("tracks_count"), "ASC") => "ORDER BY track_count ASC, sort_name COLLATE NOCASE", - (Some("tracks_count"), "DESC") => "ORDER BY track_count DESC, sort_name COLLATE NOCASE", - _ => "ORDER BY sort_name COLLATE NOCASE", + (Some("name"), "ASC") => "ORDER BY sort_name COLLATE FOLD ASC", + (Some("name"), "DESC") => "ORDER BY sort_name COLLATE FOLD DESC", + (Some("albums_count"), "ASC") => "ORDER BY album_count ASC, sort_name COLLATE FOLD", + (Some("albums_count"), "DESC") => "ORDER BY album_count DESC, sort_name COLLATE FOLD", + (Some("tracks_count"), "ASC") => "ORDER BY track_count ASC, sort_name COLLATE FOLD", + (Some("tracks_count"), "DESC") => "ORDER BY track_count DESC, sort_name COLLATE FOLD", + _ => "ORDER BY sort_name COLLATE FOLD", } } @@ -1604,7 +1599,7 @@ pub async fn search_albums( AND instr(al.canonical_title, ?) > 0 GROUP BY al.id ORDER BY (instr(al.canonical_title, ?) = 1) DESC, - al.canonical_title COLLATE NOCASE + al.canonical_title COLLATE FOLD LIMIT ? "#, ) @@ -1669,7 +1664,7 @@ pub async fn search_artists( AND instr(ar.canonical_name, ?) > 0 GROUP BY ar.id ORDER BY (instr(ar.canonical_name, ?) = 1) DESC, - ar.canonical_name COLLATE NOCASE + ar.canonical_name COLLATE FOLD LIMIT ? "#, ) @@ -1714,7 +1709,7 @@ pub async fn list_genres( LEFT JOIN artwork aw ON aw.id = g.artwork_id WHERE (? IS NULL OR t.library_id = ?) AND t.is_available = 1 GROUP BY g.id - ORDER BY g.canonical_name COLLATE NOCASE + ORDER BY g.canonical_name COLLATE FOLD "#, ) .bind(library_id) @@ -2133,7 +2128,7 @@ const FOLDER_CHILDREN_SQL: &str = r#" artwork_format FROM child GROUP BY name COLLATE NOCASE - ORDER BY name COLLATE NOCASE + ORDER BY name COLLATE FOLD "#; /// SQL for the library root that contains a path. @@ -2493,7 +2488,7 @@ pub async fn get_album_detail( JOIN track_genre tg ON tg.genre_id = g.id JOIN track t ON t.id = tg.track_id WHERE t.album_id = ? - ORDER BY g.name COLLATE NOCASE + ORDER BY g.name COLLATE FOLD "#, ) .bind(album_id) @@ -2768,7 +2763,7 @@ pub async fn get_artist_detail( WHERE al.artist_id = ?1 OR t.id IN (SELECT track_id FROM track_artist WHERE artist_id = ?1) GROUP BY al.id - ORDER BY al.year DESC, al.canonical_title COLLATE NOCASE + ORDER BY al.year DESC, al.canonical_title COLLATE FOLD "#, ) .bind(artist_id) @@ -2975,11 +2970,11 @@ pub async fn get_genre_detail( LEFT JOIN artist ar ON ar.id = t.primary_artist LEFT JOIN artwork aw ON aw.id = al.artwork_id WHERE tg.genre_id = ? AND t.is_available = 1 - ORDER BY ar.canonical_name COLLATE NOCASE, - al.canonical_title COLLATE NOCASE, + ORDER BY ar.canonical_name COLLATE FOLD, + al.canonical_title COLLATE FOLD, t.disc_number, t.track_number, - t.title COLLATE NOCASE + t.title COLLATE FOLD "#, ) .bind(genre_id) @@ -3285,7 +3280,7 @@ mod tests { ] { let clause = library_track_order_clause(Some(key), Some("asc")); assert!( - clause.starts_with("ORDER BY sort_artist COLLATE NOCASE"), + clause.starts_with("ORDER BY sort_artist COLLATE FOLD"), "{key:?} produced {clause:?}" ); assert!(!clause.contains(key) || key.is_empty(), "{key:?} leaked"); @@ -3319,7 +3314,7 @@ mod tests { assert!(track_sort_expr(key).is_some(), "{key} is not sortable"); let clause = library_track_order_clause(Some(key), Some("asc")); assert!( - !clause.starts_with("ORDER BY sort_artist COLLATE NOCASE,"), + !clause.starts_with("ORDER BY sort_artist COLLATE FOLD,"), "{key} fell through to the default" ); } @@ -3357,7 +3352,7 @@ mod tests { for key in ["artist", "year", "bitrate", "codec"] { let clause = library_track_order_clause(Some(key), Some("asc")); assert!( - clause.contains("title COLLATE NOCASE"), + clause.contains("title COLLATE FOLD"), "{key} has no tie-break: {clause}" ); } @@ -3366,7 +3361,7 @@ mod tests { // interleave their track 1s. let album = library_track_order_clause(Some("album"), Some("asc")); assert!( - album.contains("sort_artist COLLATE NOCASE, disc_number, track_number"), + album.contains("sort_artist COLLATE FOLD, disc_number, track_number"), "{album}" ); } @@ -3381,9 +3376,11 @@ mod tests { /// runs. The unified listings join the attached `app` database too, so the /// fixture attaches one and creates the single table they read. async fn pool() -> SqlitePool { - let options = SqliteConnectOptions::from_str(":memory:") - .unwrap() - .foreign_keys(true); + let options = waveflow_core::repository::sqlite::collation::register( + SqliteConnectOptions::from_str(":memory:") + .unwrap() + .foreign_keys(true), + ); let pool = SqlitePoolOptions::new() // One connection: `ATTACH` is per-connection, and a second one // would not see the attached database. @@ -4580,6 +4577,79 @@ mod tests { ); } + /// The scanner's canonical forms keep their accents, so an ASCII-only + /// collation put "Émilie Simon" after "Zazie", past the whole alphabet, + /// while the A-Z rail files her under E. + #[tokio::test] + async fn an_accented_initial_sorts_with_its_letter() { + let pool = pool().await; + seed(&pool).await; + for (id, name, canonical) in [(2, "Zazie", "zazie"), (3, "Émilie Simon", "émilie simon")] + { + sqlx::query("INSERT INTO artist (id, name, canonical_name) VALUES (?, ?, ?)") + .bind(id) + .bind(name) + .bind(canonical) + .execute(&pool) + .await + .unwrap(); + sqlx::query( + "INSERT INTO album (id, title, canonical_title, artist_id, year, is_compilation) + VALUES (?, ?, ?, ?, 2000, 0)", + ) + .bind(id) + .bind(name) + .bind(canonical) + .bind(id) + .execute(&pool) + .await + .unwrap(); + sqlx::query( + "INSERT INTO track (id, library_id, file_path, file_hash, file_size, + file_modified, title, album_id, primary_artist, + duration_ms, added_at, is_available, hlc_wall, + hlc_logical, rating_hlc_wall, rating_hlc_logical) + VALUES (?, 1, ?, ?, 1, 0, 'T', ?, ?, 1000, 500, 1, 0, 0, 0, 0)", + ) + .bind(id) + .bind(format!("/m/{id}.flac")) + .bind(format!("h{id}")) + .bind(id) + .bind(id) + .execute(&pool) + .await + .unwrap(); + sqlx::query( + "INSERT INTO track_artist (track_id, artist_id, position) VALUES (?, ?, 0)", + ) + .bind(id) + .bind(id) + .execute(&pool) + .await + .unwrap(); + } + + let names: Vec = + artists(&pool, None, None, library_artist_order_clause(None, None)) + .await + .into_iter() + .map(|row| row.1) + .collect(); + assert_eq!(names, ["Aphex Twin", "Björk", "Émilie Simon", "Zazie"]); + + let titles: Vec = albums( + &pool, + None, + None, + library_album_order_clause(Some("title"), Some("asc")), + ) + .await + .into_iter() + .map(|row| row.1) + .collect(); + assert_eq!(titles, ["Drukqs", "Émilie Simon", "Vespertine", "Zazie"]); + } + #[tokio::test] async fn the_artist_filters_behave_like_the_album_ones() { let pool = pool().await; diff --git a/src-tauri/crates/app/src/commands/inventory.rs b/src-tauri/crates/app/src/commands/inventory.rs index 8e3aa691..7079e5cd 100644 --- a/src-tauri/crates/app/src/commands/inventory.rs +++ b/src-tauri/crates/app/src/commands/inventory.rs @@ -530,9 +530,11 @@ mod tests { /// schema the app never has -- which is how `album.album_artist_id` /// survived being written down at all: the column does not exist. async fn pool() -> SqlitePool { - let options = SqliteConnectOptions::from_str(":memory:") - .unwrap() - .foreign_keys(true); + let options = waveflow_core::repository::sqlite::collation::register( + SqliteConnectOptions::from_str(":memory:") + .unwrap() + .foreign_keys(true), + ); let pool = SqlitePoolOptions::new() .max_connections(1) .connect_with(options) diff --git a/src-tauri/crates/app/src/commands/track.rs b/src-tauri/crates/app/src/commands/track.rs index 4585d554..e60bffaa 100644 --- a/src-tauri/crates/app/src/commands/track.rs +++ b/src-tauri/crates/app/src/commands/track.rs @@ -225,7 +225,7 @@ pub async fn get_track_genres( FROM track_genre tg JOIN genre g ON g.id = tg.genre_id WHERE tg.track_id = ? - ORDER BY g.name COLLATE NOCASE", + ORDER BY g.name COLLATE FOLD", ) .bind(track_id) .fetch_all(&*pool) @@ -457,11 +457,11 @@ pub async fn search_tracks_advanced( sql.push_str("ORDER BY rank\n"); } else { sql.push_str( - "ORDER BY ar.canonical_name COLLATE NOCASE,\n\ - al.canonical_title COLLATE NOCASE,\n\ + "ORDER BY ar.canonical_name COLLATE FOLD,\n\ + al.canonical_title COLLATE FOLD,\n\ t.disc_number,\n\ t.track_number,\n\ - t.title COLLATE NOCASE\n", + t.title COLLATE FOLD\n", ); } sql.push_str("LIMIT 200"); diff --git a/src-tauri/crates/app/src/db/profile_db.rs b/src-tauri/crates/app/src/db/profile_db.rs index db894cd8..c7417969 100644 --- a/src-tauri/crates/app/src/db/profile_db.rs +++ b/src-tauri/crates/app/src/db/profile_db.rs @@ -20,7 +20,9 @@ pub async fn open(path: &Path, app_db_path: &Path) -> AppResult { std::fs::create_dir_all(parent)?; } - let opts = SqliteConnectOptions::new() + // `FOLD` is how every listing sorts a name; a connection without it + // fails any query that names it. + let opts = waveflow_core::repository::sqlite::collation::register(SqliteConnectOptions::new()) .filename(path) .create_if_missing(true) .foreign_keys(true) diff --git a/src-tauri/crates/core/Cargo.toml b/src-tauri/crates/core/Cargo.toml index e081712e..4f672925 100644 --- a/src-tauri/crates/core/Cargo.toml +++ b/src-tauri/crates/core/Cargo.toml @@ -23,7 +23,9 @@ default = [] # must stay off the async runtime. Optional + feature-gated so the # postgres-only server build, which skips those modules entirely, does # not pick up a tokio dependency it has no use for. -sqlite = ["sqlx/sqlite", "sqlx/runtime-tokio", "dep:tokio"] +# `dep:unicode-normalization` decomposes accented letters for the +# `FOLD` sort collation (`repository::sqlite::collation`). +sqlite = ["sqlx/sqlite", "sqlx/runtime-tokio", "dep:tokio", "dep:unicode-normalization"] postgres = ["sqlx/postgres", "sqlx/runtime-tokio"] # Opt in to the WebAssembly Component Model plugin host. Pulls in # wasmtime + Cranelift + wasmtime-wasi + the WIT-generated bindings @@ -209,6 +211,7 @@ wasmtime = { version = "49", default-features = false, features = [ # them". wasmtime-wasi = { version = "49", default-features = false, optional = true } pinyin = { version = "0.11.0", default-features = false, features = ["plain"] } +unicode-normalization = { version = "0.1.25", optional = true } [dev-dependencies] # Sandboxed scratch directories for the smart-playlist cover render diff --git a/src-tauri/crates/core/src/metadata/name_match.rs b/src-tauri/crates/core/src/metadata/name_match.rs index d9d91313..34bf3f6c 100644 --- a/src-tauri/crates/core/src/metadata/name_match.rs +++ b/src-tauri/crates/core/src/metadata/name_match.rs @@ -82,7 +82,7 @@ pub fn normalize_name(name: &str) -> String { /// standing in an NFD-decomposed string. Covers the standard combining /// blocks; dependency-free stand-in for the `Mn` general category so we /// don't pull in `unicode-normalization` just to fold accents. -fn is_combining_mark(ch: char) -> bool { +pub(crate) fn is_combining_mark(ch: char) -> bool { matches!(ch as u32, 0x0300..=0x036F | // Combining Diacritical Marks 0x1AB0..=0x1AFF | // Combining Diacritical Marks Extended @@ -95,7 +95,7 @@ fn is_combining_mark(ch: char) -> bool { /// Map a lowercase accented Latin char to its base letter; pass anything /// else through unchanged. Covers the accents common in Western artist /// names without pulling in a Unicode-normalization dependency. -fn fold_diacritic(ch: char) -> char { +pub(crate) fn fold_diacritic(ch: char) -> char { match ch { 'á' | 'à' | 'â' | 'ä' | 'ã' | 'å' | 'ā' | 'ą' => 'a', 'ç' | 'ć' | 'č' => 'c', diff --git a/src-tauri/crates/core/src/repository/sqlite/collation.rs b/src-tauri/crates/core/src/repository/sqlite/collation.rs new file mode 100644 index 00000000..2f8f18e1 --- /dev/null +++ b/src-tauri/crates/core/src/repository/sqlite/collation.rs @@ -0,0 +1,152 @@ +//! The `FOLD` collation: how a listing sorts a name. +//! +//! `COLLATE NOCASE` folds ASCII only, so a library sorted with it puts +//! "Émilie" after "Zoé" — `É` is U+00C9, past every ASCII letter — while +//! the A-Z rail files her under E. `FOLD` compares names the way a reader +//! does: case and accents set aside, so a local "Björk" and a mirrored +//! "bjork" (the mirror's sort keys go through +//! [`normalize_name`](crate::metadata::name_match::normalize_name)) land +//! next to each other. +//! +//! The fold decomposes every letter (NFD) and drops the marks, so any +//! decomposable accent — "Ř", "Ő", "Ș", Vietnamese — sorts with its base +//! letter, not only the ones `normalize_name`'s table lists. That table +//! still runs after it for the letters NFD leaves whole ("ø"), and +//! [`fold_stroke`] covers the rest of them. +//! +//! SQLite knows nothing of it until a connection registers it, and a query +//! that names an unregistered collation fails to prepare. Every connection +//! that runs a listing must be opened through [`register`]. + +use std::cmp::Ordering; + +use sqlx::sqlite::SqliteConnectOptions; + +use unicode_normalization::UnicodeNormalization; + +use crate::metadata::name_match::{fold_diacritic, is_combining_mark}; + +/// The name queries use: `ORDER BY ar.canonical_name COLLATE FOLD`. +pub const FOLD: &str = "FOLD"; + +/// Add the [`FOLD`] collation to every connection opened with `options`. +pub fn register(options: SqliteConnectOptions) -> SqliteConnectOptions { + options.collation(FOLD, compare) +} + +/// Accents and case set aside first; then case-insensitively with the +/// accents, so "Eric" comes before "Éric"; then byte for byte. The last +/// two only break ties, and they make it a total order: two different +/// strings never compare equal, so the sort is stable between queries. +pub fn compare(a: &str, b: &str) -> Ordering { + // Most names are ASCII, and for them the fold is only the lowercasing: + // no decomposition to pay for, once per comparison of every sort. + let primary = if a.is_ascii() && b.is_ascii() { + ascii_lowered(a).cmp(ascii_lowered(b)) + } else { + folded(a).cmp(folded(b)) + }; + primary + .then_with(|| lowered(a).cmp(lowered(b))) + .then_with(|| a.cmp(b)) +} + +fn ascii_lowered(s: &str) -> impl Iterator + '_ { + s.bytes().map(|byte| byte.to_ascii_lowercase()) +} + +fn lowered(s: &str) -> impl Iterator + '_ { + s.chars().flat_map(char::to_lowercase) +} + +/// Decomposed, then stripped of its marks: "é" and "e\u{301}" both fold +/// to "e". Lazy, like [`lowered`]: a comparison allocates nothing. +fn folded(s: &str) -> impl Iterator + '_ { + s.nfd() + .filter(|ch| !is_combining_mark(*ch)) + .flat_map(char::to_lowercase) + .map(fold_diacritic) + .map(fold_stroke) +} + +/// Latin letters whose accent is part of the glyph, so NFD has nothing to +/// strip and `fold_diacritic`'s table does not list them. +fn fold_stroke(ch: char) -> char { + match ch { + 'ł' => 'l', + 'đ' => 'd', + 'ħ' => 'h', + 'ı' => 'i', + 'ŧ' => 't', + other => other, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn sorted(names: &[&str]) -> Vec { + let mut out: Vec = names.iter().map(|s| s.to_string()).collect(); + out.sort_by(|a, b| compare(a, b)); + out + } + + #[test] + fn accented_initials_sort_with_their_letter() { + assert_eq!( + sorted(&["Zoé", "Émilie", "eric", "Alain", "Ödland", "Oasis"]), + ["Alain", "Émilie", "eric", "Oasis", "Ödland", "Zoé"] + ); + } + + /// Letters `normalize_name`'s table does not list still fold: through + /// NFD for the decomposable ones, through `fold_stroke` for the rest. + #[test] + fn accents_outside_the_fold_table_sort_with_their_letter() { + assert_eq!( + sorted(&["Zeta", "Řeka", "Rosa", "Ştefan", "Łódź", "Lima", "Sara"]), + ["Lima", "Łódź", "Řeka", "Rosa", "Sara", "Ştefan", "Zeta"] + ); + } + + #[test] + fn ties_resolve_to_a_total_order() { + assert_eq!(compare("Eric", "Éric"), Ordering::Less); + assert_eq!(compare("eric", "Eric"), Ordering::Greater); + assert_eq!(compare("Éric", "Éric"), Ordering::Equal); + // Decomposed and precomposed spellings differ only past the fold. + assert_ne!(compare("E\u{301}ric", "Éric"), Ordering::Equal); + assert_eq!(sorted(&["E\u{301}ric", "Erin"]), ["E\u{301}ric", "Erin"]); + } + + #[test] + fn non_latin_scripts_keep_their_code_point_order() { + assert_eq!( + sorted(&["東京", "Abba", "Ябеда"]), + ["Abba", "Ябеда", "東京"] + ); + } + + #[tokio::test] + async fn sqlite_sorts_with_the_registered_collation() { + use sqlx::sqlite::SqlitePoolOptions; + use std::str::FromStr; + + let options = register(SqliteConnectOptions::from_str(":memory:").unwrap()); + let pool = SqlitePoolOptions::new() + .max_connections(1) + .connect_with(options) + .await + .unwrap(); + let names: Vec = sqlx::query_scalar( + "SELECT name FROM (SELECT 'Zoé' AS name UNION ALL SELECT 'Émilie' + UNION ALL SELECT 'Alain') + ORDER BY name COLLATE FOLD", + ) + .fetch_all(&pool) + .await + .unwrap(); + assert_eq!(names, ["Alain", "Émilie", "Zoé"]); + } +} diff --git a/src-tauri/crates/core/src/repository/sqlite/mod.rs b/src-tauri/crates/core/src/repository/sqlite/mod.rs index 7309a742..0983f875 100644 --- a/src-tauri/crates/core/src/repository/sqlite/mod.rs +++ b/src-tauri/crates/core/src/repository/sqlite/mod.rs @@ -9,6 +9,7 @@ //! repo.list_all().await //! ``` +pub mod collation; pub mod library; pub mod playlist; pub mod profile; diff --git a/src-tauri/crates/core/src/repository/sqlite/track.rs b/src-tauri/crates/core/src/repository/sqlite/track.rs index 2dc592a0..64fc6f38 100644 --- a/src-tauri/crates/core/src/repository/sqlite/track.rs +++ b/src-tauri/crates/core/src/repository/sqlite/track.rs @@ -74,30 +74,30 @@ fn order_clause(sort: TrackSort) -> &'static str { match (sort.column, dir) { (C::Default, _) => { - "ORDER BY ar.canonical_name COLLATE NOCASE,\n al.canonical_title COLLATE NOCASE,\n t.disc_number,\n t.track_number,\n t.title COLLATE NOCASE" + "ORDER BY ar.canonical_name COLLATE FOLD,\n al.canonical_title COLLATE FOLD,\n t.disc_number,\n t.track_number,\n t.title COLLATE FOLD" } - (C::Title, Asc) => "ORDER BY t.title COLLATE NOCASE ASC", - (C::Title, Desc) => "ORDER BY t.title COLLATE NOCASE DESC", + (C::Title, Asc) => "ORDER BY t.title COLLATE FOLD ASC", + (C::Title, Desc) => "ORDER BY t.title COLLATE FOLD DESC", (C::Artist, Asc) => { - "ORDER BY ar.canonical_name COLLATE NOCASE ASC, t.title COLLATE NOCASE" + "ORDER BY ar.canonical_name COLLATE FOLD ASC, t.title COLLATE FOLD" } (C::Artist, Desc) => { - "ORDER BY ar.canonical_name COLLATE NOCASE DESC, t.title COLLATE NOCASE" + "ORDER BY ar.canonical_name COLLATE FOLD DESC, t.title COLLATE FOLD" } (C::Album, Asc) => { - "ORDER BY al.canonical_title COLLATE NOCASE ASC, t.disc_number, t.track_number" + "ORDER BY al.canonical_title COLLATE FOLD ASC, t.disc_number, t.track_number" } (C::Album, Desc) => { - "ORDER BY al.canonical_title COLLATE NOCASE DESC, t.disc_number, t.track_number" + "ORDER BY al.canonical_title COLLATE FOLD DESC, t.disc_number, t.track_number" } (C::DurationMs, Asc) => "ORDER BY t.duration_ms ASC", (C::DurationMs, Desc) => "ORDER BY t.duration_ms DESC", - (C::Year, Asc) => "ORDER BY t.year ASC, t.title COLLATE NOCASE", - (C::Year, Desc) => "ORDER BY t.year DESC, t.title COLLATE NOCASE", + (C::Year, Asc) => "ORDER BY t.year ASC, t.title COLLATE FOLD", + (C::Year, Desc) => "ORDER BY t.year DESC, t.title COLLATE FOLD", (C::AddedAt, Asc) => "ORDER BY t.added_at ASC", (C::AddedAt, Desc) => "ORDER BY t.added_at DESC", - (C::Rating, Asc) => "ORDER BY t.rating ASC, t.title COLLATE NOCASE", - (C::Rating, Desc) => "ORDER BY t.rating DESC, t.title COLLATE NOCASE", + (C::Rating, Asc) => "ORDER BY t.rating ASC, t.title COLLATE FOLD", + (C::Rating, Desc) => "ORDER BY t.rating DESC, t.title COLLATE FOLD", } } @@ -207,7 +207,7 @@ impl TrackRepository for SqliteTrackRepository { sql.push_str(LIKE_TERM_CLAUSE); sql.push('\n'); } - sql.push_str("ORDER BY t.title COLLATE NOCASE\nLIMIT ?"); + sql.push_str("ORDER BY t.title COLLATE FOLD\nLIMIT ?"); let mut q = sqlx::query_as::<_, TrackRow>(sqlx::AssertSqlSafe(sql)); for p in &patterns { @@ -225,17 +225,17 @@ impl TrackRepository for SqliteTrackRepository { let (sql, id) = match source { TrackSource::Folder(id) => ( "SELECT id FROM track WHERE folder_id = ? AND is_available = 1 - ORDER BY disc_number, track_number, title COLLATE NOCASE", + ORDER BY disc_number, track_number, title COLLATE FOLD", id, ), TrackSource::Album(id) => ( "SELECT id FROM track WHERE album_id = ? AND is_available = 1 - ORDER BY disc_number, track_number, title COLLATE NOCASE", + ORDER BY disc_number, track_number, title COLLATE FOLD", id, ), TrackSource::Artist(id) => ( "SELECT id FROM track WHERE primary_artist = ? AND is_available = 1 - ORDER BY title COLLATE NOCASE", + ORDER BY title COLLATE FOLD", id, ), }; @@ -301,6 +301,7 @@ impl TrackRepository for SqliteTrackRepository { mod tests { use super::*; use crate::search::plan_search; + use std::str::FromStr; /// Enough of the profile schema for `SELECT_TRACK_ROW` to resolve, /// plus the FTS index the search actually reads. @@ -312,7 +313,10 @@ mod tests { /// `waveflow-core` (same constraint the scanner fixtures note), so /// this is a copy — if the tokenizer changes there, change it here. async fn fixture_pool() -> SqlitePool { - let pool = SqlitePool::connect("sqlite::memory:").await.unwrap(); + let options = super::super::collation::register( + sqlx::sqlite::SqliteConnectOptions::from_str("sqlite::memory:").unwrap(), + ); + let pool = SqlitePool::connect_with(options).await.unwrap(); sqlx::raw_sql( "PRAGMA foreign_keys = ON; CREATE TABLE artwork (