Repository navigation
fix: preserve unchanged collated indexes - #50
Conversation
There was a problem hiding this comment.
Pull request overview
Prevents repeated rebuilding of collated MongoDB indexes during synchronization.
Changes:
- Normalizes stored collations before comparison.
- Adds unit and ignored integration tests for collation synchronization.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/index.rs |
Adds collation normalization and unit tests. |
tests/indexes.rs |
Adds MongoDB integration tests for collated indexes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
tests/indexes.rs:306
- Add the missing auxiliary verb so this documentation sentence is grammatical.
/// Documents the server behaviour the sync test depends on, and kept separate from it so a failure
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/index.rs:568
explainrequires authorization to execute its underlyingfind. This adds a new read privilege requirement tosync_indexes, so least-privilege credentials that can list/create/drop indexes but cannot read documents will now fail withUnauthorizedwhenever a matching collated index exists. Please avoid a find-based normalization mechanism or explicitly provide a compatible fallback/contract for these callers.
"explain": { "find": collection, "filter": {}, "collation": collation },
src/index.rs:577
- These
.ok()calls collapse a malformed or changedexplainresponse into the sameNoneused forlocale: "simple". For any non-simple declaration, a missingqueryPlanner.collationtherefore strips the requested collation; if the existing index is collation-less, it can then be incorrectly marked synchronized instead of returning an error. Only treat an absent collation as valid for the simple locale and fail closed otherwise.
Ok(ret
.get_document("queryPlanner")
.ok()
.and_then(|planner| planner.get_document("collation").ok().cloned()))
src/index.rs:409
- The stated
versioncarve-out is not implemented here: the expansion keeps the current server's ICUversion, whileexisting_indexkeeps the version recorded when that index was created, anddoc_are_eqcompares them exactly. After an ICU/server upgrade this can still drop an otherwise unchanged collated index solely because those server-owned versions differ. Removeversionfrom both server-produced collations before comparing them.
let expanded = expanded_collation(
db,
CollConf::collection_name(),
&declared_collation,
)
There was a problem hiding this comment.
Thank you. Indeed, we should compare the compact collation against the expanded one, otherwise we end up dropping and rebuilding the index at every restart. The explain call role is to expand the declaration the same way the server would, so we can do the right comparison. The idea looks sound to me!
I’m attaching a few findings below.
Benoît Cortier (CBenoit)
left a comment
There was a problem hiding this comment.
Another nitpick: In this repository we are following the conventional commit specification: https://www.conventionalcommits.org/en/v1.0.0/
Roughly, the shape of the final, squashed commit should be something like:
fix: stop rebuilding collated indexes on every sync_indexes call
<some details>
Issue: HB-8770
Where the ticket/issue key is attached as a footer instead, and the PR title should match the commit summary.
MongoDB expands a collation when it creates an index, so listIndexes returns
every ICU field plus a version of its own. doc_are_eq compares the declared
index against that response with exact document equality, stripping only "ns"
and "v", so a declaration of {"locale": "en", "strength": 2} could never match
the index it had just created. Every sync_indexes call concluded the index had
changed and dropped and recreated it.
Observed against MongoDB 7 (ICU 57.1): two consecutive sync_indexes calls with
nothing changed in between moved $indexStats accesses.since, confirming the
index was destroyed and rebuilt by a no-op sync. For a caller that syncs indexes
at startup, that is an index rebuild on every process start plus a window on
each start where the collection is unindexed.
Declaring the expanded collation is not a workaround, since version is chosen by
the server.
Fixed by reducing the stored collation to the fields that were actually declared
before comparing, which mirrors the carve-outs sync_indexes already makes for
server-supplied "ns"/"v" and for text-index weights. Only server-filled fields
are discarded: changing, adding or removing a declared field still compares
unequal and still rebuilds the index, each covered by a unit test that needs no
running MongoDB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Filtering the stored collation down to the declared keys silently accepted the
removal of a single option. Dropping `strength: 2` from a declaration of
{"locale": "en", "strength": 2} left the stored index on strength 2 while the
declaration now means MongoDB's default of 3, and the two compared equal, so the
index was never rebuilt.
An undeclared field is only noise when it holds the value the server would apply
again next time, so keep it whenever it holds anything else and let doc_are_eq
see the difference. Fields with no documented default, including the server-owned
`version`, stay ignored so a field added by a future server release cannot
reintroduce the rebuild-on-every-sync behaviour.
Also gives each collated integration test its own collection. Both dropped
`collated_sync` on the way in, and libtest runs ignored tests concurrently, so
either could have pulled the collection out from under the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two more declarations that never converged, both verified against MongoDB 7 by syncing twice and watching $indexStats accesses.since move. locale "simple" asks for plain binary comparison, which the server records by storing no collation on the index at all. The declaration carried a collation key the stored index lacked, so doc_are_eq saw a difference every run. Dropped from the declaration so it compares against what the server actually keeps, while moving an existing collated index to the simple locale is still a real change and still rebuilds. Collation values were compared by BSON variant rather than by number, so a caller writing strength as an i64 produced Int64(2) against the server's Int32(2) and rebuilt forever. Plausible whenever the value comes from a variable rather than a literal. Integers are now canonicalized on both sides, which also protects the default comparison from a server that reports strength as an i64. Uses is_ok_and rather than comparing Result with ==, since the bson 3 error type is not PartialEq and the latter does not build under that feature set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collation defaults are locale-dependent, so the table of "en" defaults was wrong for every locale that deviates and rebuilt those indexes on every sync — the very bug this branch set out to fix. Verified against MongoDB 7: fr_CA expands with backwards true, da with caseFirst "upper", th with alternate "shifted" and normalization true, vi with normalization true. A locale-aware table would only move the boundary, since it is still wrong for whatever locale it omits and for whatever ICU changes between releases. The server already knows the answer: an explain on a trivial find reports the same expansion it would store, including its own ICU version, and reports no collation at all for locale "simple" — exactly matching how it stores that request. So the declared collation is replaced by the server's expansion of it before comparing, and both sides then hold a document the server produced. This drops the defaults table, the simple-locale special case and the integer canonicalization, all of which existed to approximate what the server now supplies directly. Costs one explain per collated index per sync, on a path that already runs listIndexes and only at startup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MongoDB expands a collation when it creates an index, so listIndexes returns
every ICU field plus a version of its own. doc_are_eq compared the declaration
against that with exact document equality, stripping only "ns" and "v", so a
declaration of {"locale": "en", "strength": 2} could never match the index it had
just created and every call concluded the index had changed. For a caller that
syncs indexes at startup that means a rebuild on every process start plus a
window on each start where the collection is unindexed.
Guessing the fields the server fills in does not work: the values are
locale-dependent. fr_CA expands with backwards true, da with caseFirst "upper",
th with alternate "shifted" and normalization true, vi with normalization true,
so a table of defaults is wrong for every locale it omits. The server already
knows the answer, and an explain on a trivial find reports the very expansion it
would store, including its own ICU version. The declaration is therefore replaced
by that expansion before comparing, and both sides then hold a document the
server produced. Any real change to the declaration expands differently and is
still caught.
Reading the expansion is separated from running the command so every response
shape is covered without a live deployment:
- A standalone reply carries the collation on queryPlanner.
- mongos carries no collation there and reports one entry per shard under
winningPlan.shards. Each entry is itself a planner, with namespace, collation
and winningPlan directly on it rather than a nested queryPlanner. A nested one
is accepted too. Shards must agree.
- No collation reported means the server stores the index without one, which is
how it answers locale "simple".
- Any shape this cannot read is an error rather than "no collation". Confusing
the two would drop the collation from the declaration, compare it equal to an
uncollated index, and report success without building the index that was asked
for.
explain needs the find privilege, which index management alone did not require.
An Unauthorized failure therefore falls back to comparing the declaration as
written: synchronization still succeeds and the index is still correct, but it is
rebuilt on every call. sync_indexes documents this so a least-privilege role that
cannot read documents keeps working, at that cost. Any other error propagates.
Verified against MongoDB 8.2. Two consecutive syncs left $indexStats
accesses.since unchanged for en, fr_CA, th and simple, and moved it when the
declaration changed. Checked through a real mongos over a sharded collection,
which is what revealed the per-shard shape, and against a role holding index
management without find, which churns while a role holding both converges.
The unit tests need no running MongoDB, unlike everything in tests/indexes.rs,
where the whole suite is #[ignore]d and CI has no mongo service — the reason this
went unnoticed.
Issue: HB-8770
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Handle unfamiliar explain replies safely and verify the missing-find fallback with an authenticated restricted role. Issue: HB-8770 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
19788b8 to
34db6fb
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/index.rs:618
- A planner with no top-level collation is treated as
NonewheneverwinningPlan.shardscannot be read. That includes unrecognized replies such as{"queryPlanner": {}}and malformed sharded replies whereshardsis not an array.apply_expanded_collationthen removes the declared collation, so an existing uncollated index can compare equal and synchronization can incorrectly leave it uncollated. Validate a recognizable standalone planner before returningNone, and report every other shape as unrecognized.
return Ok(Some(collation.clone()));
}
if planner.contains_key("winningPlan") || planner.contains_key("namespace") {
Route all ignored integration tests through MONGODB_URI so they authenticate against CI's MongoDB service. Issue: HB-8770 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/index.rs:606
- A successful reply containing any
queryPlannerobject is currently treated as a recognized “no collation” response wheneverwinningPlan.shardsis absent. For example, an unfamiliar{ queryPlanner: {} }shape becomesExpanded(None), causing the declared collation to be removed before comparison; if the stored index is uncollated, synchronization can then incorrectly leave it unchanged. Validate the standalone planner shape asshard_collationdoes so unfamiliar responses take the safe rebuild fallback.
match sharded {
Some(shards) => collation_agreed_across_shards(shards),
// A standalone planner that reports no collation: the index is stored without one.
None => Ok(None),
Compare only server-expanded collations without field order so ordered BSON index options still trigger a rebuild. Issue: HB-8770 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Expand declared collations through MongoDB before comparing index definitions, so equivalent collated indexes remain intact across syncs.
Fall back safely when expansion is unavailable or unrecognized.
Issue: HB-8770