Skip to content

fix: preserve unchanged collated indexes - #50

Merged
Benoît Cortier (CBenoit) merged 8 commits into
masterfrom
fix/hb-8770-collated-index-sync
Aug 7, 2026
Merged

Benoît Cortier (CBenoit) merged 8 commits into
masterfrom
fix/hb-8770-collated-index-sync

Conversation

@B-E-A-F

@B-E-A-F Bea (B-E-A-F) commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Expand declared collations through MongoDB before comparing index definitions, so equivalent collated indexes remain intact across syncs.

Fall back safely when expansion is unavailable or unrecognized.

Issue: HB-8770

@B-E-A-F Bea (B-E-A-F) changed the title fix: stop rebuilding collated indexes on every sync_indexes call [HB-8770] stop rebuilding collated indexes on every sync_indexes call Aug 5, 2026
@B-E-A-F
Bea (B-E-A-F) requested a balanced review from Copilot August 5, 2026 16:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents repeated rebuilding of collated MongoDB indexes during synchronization.

Changes:

  • Normalizes stored collations before comparison.
  • Adds unit and ignored integration tests for collation synchronization.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/index.rs Adds collation normalization and unit tests.
tests/indexes.rs Adds MongoDB integration tests for collated indexes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/index.rs Outdated
Comment thread tests/indexes.rs Outdated

@B-E-A-F Bea (B-E-A-F) left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(removed)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

tests/indexes.rs:306

  • Add the missing auxiliary verb so this documentation sentence is grammatical.
/// Documents the server behaviour the sync test depends on, and kept separate from it so a failure

Comment thread src/index.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/index.rs:568

  • explain requires authorization to execute its underlying find. This adds a new read privilege requirement to sync_indexes, so least-privilege credentials that can list/create/drop indexes but cannot read documents will now fail with Unauthorized whenever a matching collated index exists. Please avoid a find-based normalization mechanism or explicitly provide a compatible fallback/contract for these callers.
            "explain": { "find": collection, "filter": {}, "collation": collation },

src/index.rs:577

  • These .ok() calls collapse a malformed or changed explain response into the same None used for locale: "simple". For any non-simple declaration, a missing queryPlanner.collation therefore strips the requested collation; if the existing index is collation-less, it can then be incorrectly marked synchronized instead of returning an error. Only treat an absent collation as valid for the simple locale and fail closed otherwise.
    Ok(ret
        .get_document("queryPlanner")
        .ok()
        .and_then(|planner| planner.get_document("collation").ok().cloned()))

src/index.rs:409

  • The stated version carve-out is not implemented here: the expansion keeps the current server's ICU version, while existing_index keeps the version recorded when that index was created, and doc_are_eq compares them exactly. After an ICU/server upgrade this can still drop an otherwise unchanged collated index solely because those server-owned versions differ. Remove version from both server-produced collations before comparing them.
                        let expanded = expanded_collation(
                            db,
                            CollConf::collection_name(),
                            &declared_collation,
                        )

@CBenoit Benoît Cortier (CBenoit) left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Indeed, we should compare the compact collation against the expanded one, otherwise we end up dropping and rebuilding the index at every restart. The explain call role is to expand the declaration the same way the server would, so we can do the right comparison. The idea looks sound to me!

I’m attaching a few findings below.

Comment thread src/index.rs Outdated
Comment thread tests/indexes.rs
Comment thread src/index.rs Outdated
Comment thread src/index.rs

@CBenoit Benoît Cortier (CBenoit) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another nitpick: In this repository we are following the conventional commit specification: https://www.conventionalcommits.org/en/v1.0.0/

Roughly, the shape of the final, squashed commit should be something like:

fix: stop rebuilding collated indexes on every sync_indexes call

<some details>

Issue: HB-8770

Where the ticket/issue key is attached as a footer instead, and the PR title should match the commit summary.

Bea (B-E-A-F) and others added 6 commits August 7, 2026 17:23
MongoDB expands a collation when it creates an index, so listIndexes returns
every ICU field plus a version of its own. doc_are_eq compares the declared
index against that response with exact document equality, stripping only "ns"
and "v", so a declaration of {"locale": "en", "strength": 2} could never match
the index it had just created. Every sync_indexes call concluded the index had
changed and dropped and recreated it.

Observed against MongoDB 7 (ICU 57.1): two consecutive sync_indexes calls with
nothing changed in between moved $indexStats accesses.since, confirming the
index was destroyed and rebuilt by a no-op sync. For a caller that syncs indexes
at startup, that is an index rebuild on every process start plus a window on
each start where the collection is unindexed.

Declaring the expanded collation is not a workaround, since version is chosen by
the server.

Fixed by reducing the stored collation to the fields that were actually declared
before comparing, which mirrors the carve-outs sync_indexes already makes for
server-supplied "ns"/"v" and for text-index weights. Only server-filled fields
are discarded: changing, adding or removing a declared field still compares
unequal and still rebuilds the index, each covered by a unit test that needs no
running MongoDB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Filtering the stored collation down to the declared keys silently accepted the
removal of a single option. Dropping `strength: 2` from a declaration of
{"locale": "en", "strength": 2} left the stored index on strength 2 while the
declaration now means MongoDB's default of 3, and the two compared equal, so the
index was never rebuilt.

An undeclared field is only noise when it holds the value the server would apply
again next time, so keep it whenever it holds anything else and let doc_are_eq
see the difference. Fields with no documented default, including the server-owned
`version`, stay ignored so a field added by a future server release cannot
reintroduce the rebuild-on-every-sync behaviour.

Also gives each collated integration test its own collection. Both dropped
`collated_sync` on the way in, and libtest runs ignored tests concurrently, so
either could have pulled the collection out from under the other.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two more declarations that never converged, both verified against MongoDB 7 by
syncing twice and watching $indexStats accesses.since move.

locale "simple" asks for plain binary comparison, which the server records by
storing no collation on the index at all. The declaration carried a collation key
the stored index lacked, so doc_are_eq saw a difference every run. Dropped from
the declaration so it compares against what the server actually keeps, while
moving an existing collated index to the simple locale is still a real change and
still rebuilds.

Collation values were compared by BSON variant rather than by number, so a caller
writing strength as an i64 produced Int64(2) against the server's Int32(2) and
rebuilt forever. Plausible whenever the value comes from a variable rather than a
literal. Integers are now canonicalized on both sides, which also protects the
default comparison from a server that reports strength as an i64.

Uses is_ok_and rather than comparing Result with ==, since the bson 3 error type
is not PartialEq and the latter does not build under that feature set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collation defaults are locale-dependent, so the table of "en" defaults was wrong
for every locale that deviates and rebuilt those indexes on every sync — the very
bug this branch set out to fix. Verified against MongoDB 7: fr_CA expands with
backwards true, da with caseFirst "upper", th with alternate "shifted" and
normalization true, vi with normalization true.

A locale-aware table would only move the boundary, since it is still wrong for
whatever locale it omits and for whatever ICU changes between releases. The server
already knows the answer: an explain on a trivial find reports the same expansion
it would store, including its own ICU version, and reports no collation at all for
locale "simple" — exactly matching how it stores that request.

So the declared collation is replaced by the server's expansion of it before
comparing, and both sides then hold a document the server produced. This drops the
defaults table, the simple-locale special case and the integer canonicalization,
all of which existed to approximate what the server now supplies directly.

Costs one explain per collated index per sync, on a path that already runs
listIndexes and only at startup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MongoDB expands a collation when it creates an index, so listIndexes returns
every ICU field plus a version of its own. doc_are_eq compared the declaration
against that with exact document equality, stripping only "ns" and "v", so a
declaration of {"locale": "en", "strength": 2} could never match the index it had
just created and every call concluded the index had changed. For a caller that
syncs indexes at startup that means a rebuild on every process start plus a
window on each start where the collection is unindexed.

Guessing the fields the server fills in does not work: the values are
locale-dependent. fr_CA expands with backwards true, da with caseFirst "upper",
th with alternate "shifted" and normalization true, vi with normalization true,
so a table of defaults is wrong for every locale it omits. The server already
knows the answer, and an explain on a trivial find reports the very expansion it
would store, including its own ICU version. The declaration is therefore replaced
by that expansion before comparing, and both sides then hold a document the
server produced. Any real change to the declaration expands differently and is
still caught.

Reading the expansion is separated from running the command so every response
shape is covered without a live deployment:

- A standalone reply carries the collation on queryPlanner.
- mongos carries no collation there and reports one entry per shard under
  winningPlan.shards. Each entry is itself a planner, with namespace, collation
  and winningPlan directly on it rather than a nested queryPlanner. A nested one
  is accepted too. Shards must agree.
- No collation reported means the server stores the index without one, which is
  how it answers locale "simple".
- Any shape this cannot read is an error rather than "no collation". Confusing
  the two would drop the collation from the declaration, compare it equal to an
  uncollated index, and report success without building the index that was asked
  for.

explain needs the find privilege, which index management alone did not require.
An Unauthorized failure therefore falls back to comparing the declaration as
written: synchronization still succeeds and the index is still correct, but it is
rebuilt on every call. sync_indexes documents this so a least-privilege role that
cannot read documents keeps working, at that cost. Any other error propagates.

Verified against MongoDB 8.2. Two consecutive syncs left $indexStats
accesses.since unchanged for en, fr_CA, th and simple, and moved it when the
declaration changed. Checked through a real mongos over a sharded collection,
which is what revealed the per-shard shape, and against a role holding index
management without find, which churns while a role holding both converges.

The unit tests need no running MongoDB, unlike everything in tests/indexes.rs,
where the whole suite is #[ignore]d and CI has no mongo service — the reason this
went unnoticed.

Issue: HB-8770

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Handle unfamiliar explain replies safely and verify the missing-find fallback with an authenticated restricted role.

Issue: HB-8770

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@CBenoit Benoît Cortier (CBenoit) changed the title [HB-8770] stop rebuilding collated indexes on every sync_indexes call fix: preserve unchanged collated indexes Aug 7, 2026
@CBenoit
Benoît Cortier (CBenoit) requested a balanced review from Copilot August 7, 2026 09:36
@CBenoit
Benoît Cortier (CBenoit) enabled auto-merge (squash) August 7, 2026 09:36
@CBenoit
Benoît Cortier (CBenoit) force-pushed the fix/hb-8770-collated-index-sync branch from 19788b8 to 34db6fb Compare August 7, 2026 09:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/index.rs:618

  • A planner with no top-level collation is treated as None whenever winningPlan.shards cannot be read. That includes unrecognized replies such as {"queryPlanner": {}} and malformed sharded replies where shards is not an array. apply_expanded_collation then removes the declared collation, so an existing uncollated index can compare equal and synchronization can incorrectly leave it uncollated. Validate a recognizable standalone planner before returning None, and report every other shape as unrecognized.
        return Ok(Some(collation.clone()));
    }

    if planner.contains_key("winningPlan") || planner.contains_key("namespace") {

Comment thread src/index.rs
Route all ignored integration tests through MONGODB_URI so they authenticate against CI's MongoDB service.

Issue: HB-8770

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/index.rs:606

  • A successful reply containing any queryPlanner object is currently treated as a recognized “no collation” response whenever winningPlan.shards is absent. For example, an unfamiliar { queryPlanner: {} } shape becomes Expanded(None), causing the declared collation to be removed before comparison; if the stored index is uncollated, synchronization can then incorrectly leave it unchanged. Validate the standalone planner shape as shard_collation does so unfamiliar responses take the safe rebuild fallback.
    match sharded {
        Some(shards) => collation_agreed_across_shards(shards),
        // A standalone planner that reports no collation: the index is stored without one.
        None => Ok(None),

Comment thread src/index.rs Outdated
Compare only server-expanded collations without field order so ordered BSON index options still trigger a rebuild.

Issue: HB-8770

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@CBenoit Benoît Cortier (CBenoit) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@CBenoit
Benoît Cortier (CBenoit) merged commit 088c996 into master Aug 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants