fix(core): drive-qualify aggregate rollup and duplicate keys#575
Merged
Conversation
Multi-drive aggregations merged per-drive identifiers as if they were global, corrupting every rollup and extension-keyed result: - Rollup group keys were bare per-drive record indices. The cross-drive merge collided buckets from different drives and finalize resolved every key against the FIRST drive's records, producing malformed folder names (or record_N fallbacks) from the top_folders preset. Keys now carry the drive ordinal (high 32 bits) and resolve against the drive they came from. - Duplicate composite keys pushed the raw per-drive extension_id for ext key fields: the same extension never grouped across drives while different extensions with colliding intern ids grouped falsely. The key now hashes the interned (lowercased) extension string. - Nested rollup sub-aggregations finalized without the cross-drive ExtensionMap, resolving canonical extension ids through a legacy per-drive fallback. The map is now a required parameter through the feed and finalize chains and both dead fallbacks are deleted. - Ancestor/drilldown rollups applied their per-drive record index to every scanned drive. RollupMode::Ancestor now names its drive (rollup:ancestor,record=N,drive=C); records on other drives are skipped, including in nested sub-accumulators. The wire spec gained an optional backward-compatible drive slot and the daemon rejects ancestor specs without one. The duplicates.rs test module moved to duplicates/tests.rs (mirroring the compact_cache/tests.rs extraction) to stay under the 800 LOC file policy without adding an exception. Regression tests cover the multi-drive top_folders keys, cross-drive extension duplicate grouping, ancestor drive scoping, parser syntax and error paths, and wire conversion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Multi-drive aggregations merged per-drive identifiers as if they were global, corrupting rollup and extension-keyed results. Found live via the MCP
top_folderspreset returning malformed, unusable keys on a multi-drive host.Fixes
Rollup keys (top_folders and all path/drive rollups). Group keys were bare per-drive record indices. The cross-drive merge collided buckets from different drives, and finalize resolved every key against the first drive's records — malformed folder names or
record_Nfallbacks, with one drive's byte totals silently folded into another's buckets. Keys now carry the drive ordinal (high 32 bits) and resolve against the drive they came from.Duplicate composite keys.
duplicates:size+extpushed the raw per-driveextension_id: the same extension never grouped across drives, while different extensions with colliding intern ids grouped falsely. The key now hashes the interned (lowercased) extension string.Nested rollup sub-aggregations.
rollup:...,sub=terms:extfinalized without the cross-driveExtensionMap, resolving canonical ids through a legacy per-drive fallback (wrong extension names in sub-buckets). The map is now a required parameter through the feed and finalize chains; both dead fallbacks are deleted.Ancestor/drilldown rollups. The per-drive
record_idxwas applied to every scanned drive, bucketing other drives' records against arbitrary unrelated folders.RollupMode::Ancestornow names its drive (rollup:ancestor,record=N,drive=C); out-of-scope records are skipped, including in nested sub-accumulators. The wire spec gained an optional, backward-compatibledriveslot; the daemon rejects ancestor specs without one (typedWireSpecError::AncestorDriveMissing).Structure
duplicates.rstest module moved toduplicates/tests.rs(mirrors thecompact_cache/tests.rsextraction) to stay under the 800 LOC policy without adding an exception.Tests
All regression tests were verified to fail against the pre-fix code:
top_folders_multi_drive_keys_resolve_per_drive— pre-fix, drive D's folder vanished entirely and its bytes were credited toC:\Projectsextension_key_groups_across_drives_by_name_not_intern_id— pre-fix, the false avi/mp4 id-collision group survived while the real cross-drive jpg pair was missedancestor_rollup_scoped_to_named_drive— drilldown intoC:\Projectsbuckets only drive C records with exact byte totalsdrive=syntax and both error paths; wire conversion accept/reject tests; rollup key encode/decode round-trip and out-of-range-ordinal fallbackGates: full workspace suite (2493 tests), lint-prod, lint-tests, rustdoc (
--document-private-items --locked), doc-tests, and xwin Windows clippy all green; the full pre-push bundle passed on push.