feat(mft): extended $SI fields, parse status, and macOS target-cpu fix#574
Merged
Conversation
`emit_build_env` is behind `#[cfg(feature = "build")]`, so the two [`emit_build_env`] links in uffs-version's docs do not resolve when the crate is documented without that feature. The pre-push rustdoc gate runs `cargo doc --workspace --no-deps --document-private-items --locked` with `-D rustdoc::broken-intra-doc-links`, so this failed on main. Demote both to plain code spans that name the feature gate. The workspace rustdoc gate now passes.
The NTFS 3.0+ 72-byte $STANDARD_INFORMATION carries four fields past the timestamps that the parser read into `StandardInformationExtended` and then dropped: `quota_charged`, `max_versions`, `version_number`, and `class_id`. Downstream consumers that build a full forensic record had to re-read them from the raw bytes with their own hard-coded offset table. Add them to `ExtendedStandardInfo` and populate them in the NTFS 3.0+ branch of `parse_standard_info_full`, the single source of truth for $SI parsing. The NTFS 1.2 branch is unchanged: that 36-byte layout has no such fields, and `..from_attributes(...)` correctly leaves them zero. Costs nothing persistent. `ExtendedStandardInfo` is a stack temporary in both production parsers, converted immediately via `StandardInfo::from_extended` and dropped; the 56-byte `StandardInfo` that lands in `MftIndex` is untouched, so no index, cache, or on-disk format changes. The values are already materialized by the zerocopy cast, so this is three u32 moves and one u64. Additive and semver-minor: every construction site uses `::default()` or `..from_attributes(...)`, so struct-update syntax covers the new fields.
`parse_record_to_fragment` hand-rolled its own $STANDARD_INFORMATION read and unconditionally parsed the 36-byte NTFS 1.2 layout, ignoring the attribute's real value_length. On any NTFS 3.0+ volume — i.e. all of them — it therefore returned usn, security_id and owner_id as zero while looking like it had parsed the attribute successfully. Route it through `parse_standard_info_full`, the same single source of truth the two live parsers already use, so all three paths agree. The directory flag is applied after the attribute loop, so moving the read does not disturb it. Malformed $SI now leaves `std_info` at its default and continues the loop instead of aborting the whole attribute walk, matching `unified::process_record`. The parser is deprecated and has no callers in the workspace, so this is latent rather than user-visible today; it is kept for a planned carving consumer, which makes correctness now cheaper than archaeology later. Also replaces the `too_many_lines` expect reason, which claimed this was a "performance-critical hot path" — false since the fragment-merge rewrite, and precisely why the bug went unexamined this long. Adds a regression pin on the fragment path alongside the existing two-parser one; verified to fail (usn 0 != 66) against the pre-fix code.
… the helper The existing pins assert `parse_standard_info_full` directly, but the public API consumers actually call is `parse_record_full`, which reaches those fields through two hops: a local `ExtendedStandardInfo` in `full.rs` that is later moved onto `ParsedRecord::std_info`. A refactor giving `full.rs` its own $SI read would keep the helper-level tests green while silently zeroing the fields for every external caller. Assert quota_charged/max_versions/version_number/class_id — plus the pre-existing usn/security_id/owner_id, which ride the same two hops — on a `ParsedRecord` returned by `parse_record_full`. Verified to fail when the second hop is broken (quota_charged 0 != 65536). The addition pushed parse/tests.rs to 839 LOC, past the 800-LOC policy limit. Split rather than granted an exception: tests.rs becomes tests/mod.rs (698 LOC, helpers and everything else) plus tests/std_info.rs (152 LOC, all three $STANDARD_INFORMATION pins). Also drops the now-false `single_call_fn` expectation on `create_file_name_value`, which the new test makes a second caller of.
….rs out The comment above the forensic-flag accessors listed only bits 0-2 and stopped, which reads as "bit 3 onwards is free". It is not: bits 3-5 are has_default_data, has_i30_stream and is_unified, written by their own setters, which is why `set_forensic_flags` masks them through with `& 0b11_1000`. Bit 3 feeds `compute_tree_metrics`, where it separates "empty $DATA" from "no $DATA", so a new flag landing there would surface as wrong treesize/tree_allocated far from the cause. Only bits 6-7 are free. The table is now complete and says so. types.rs sat at exactly 800 LOC, so the full comment pushed it over the policy limit. Rather than trim the docs to fit or take a size exception, lift the "Safe Casting Helpers" section — 15 numeric conversions with no dependency on the record structs, and no callers inside types.rs — into a new `index/convert.rs` (211 LOC), leaving types.rs at 610 with room to document itself. Re-exported from `index.rs` under the same public paths, so no call site outside the crate changes; `index/base.rs` was the one in-crate user of a `types::`-qualified path and now points at `convert`.
When $SI could not be decoded, the parser returned silently and left
`ExtendedStandardInfo` at its defaults — every timestamp `0`, i.e. FILETIME
1601-01-01, which is a value a real record can legitimately hold. Nothing
recorded that the attribute had failed, so a consumer building a forensic
record could only report the default as fact. `is_corrupt` does not cover
this: it is set solely by the caller on fixup/USA failure, and there was no
attribute-level damage representation anywhere in the crate.
`parse_standard_info_full` now returns `StdInfoParse`:
V30 decoded as the 72-byte NTFS 3.0+ layout
V12 decoded as the 36-byte NTFS 1.2 layout (missing extended
fields are correct, not damage)
Malformed present but undecodable — resident header past the end of the
record, declared length under the 36-byte minimum, or payload
overrunning the buffer
Absent no $SI attribute at all; unusual, but not evidence of damage,
and deliberately distinct from Malformed
Surfaced on `ParsedRecord::std_info_parse` so it reaches the public API
(`parse_record_full`, `parse_record_forensic`), which is where an external
index reads from. Deliberately not `#[must_use]`: the bulk pipelines
(unified, direct-index, fragment) have nowhere to record it, and dropping
it there is the correct choice, not an oversight worth a lint.
Costs nothing measurable. `ParsedRecord` stays at 256 bytes — the field
lands in existing padding — and `StdInfoParse` is 1 byte returned in a
register. The variant is just which branch the parser already took, so no
new work is done; the three bulk callers discard it and the value is dead.
No index, cache, or on-disk format change.
Groundwork only: representation (columns, DataFrame projection, the
daemon's shmem row) is deliberately left to the consumer, which builds its
own index. Closes #573.
`-C target-cpu=native` resolves to bare `generic` on every GitHub macOS runner: they report `Apple M1 (Virtual)`, and LLVM's host detection cannot identify the virtualised CPU, so it falls back to the ARMv8-A baseline. Measured on the runners themselves — `native` yields exactly one feature, `neon`, identical to `-C target-cpu=generic`. Two consequences, one loud and one silent: - Loud: `ring` asserts at compile time that `aes` and `sha2` are statically present on aarch64-apple-darwin, so it refuses to build. This is the macOS Nightly Canary failure (#553), red weekly since at least 2026-06-08. It was diagnosed as an upstream `ring 0.17.14` regression to wait out; it is neither upstream nor a regression. Reproduced locally on the pinned toolchain with `-C target-cpu=generic`: byte-identical E0080s at darwin.rs:44 and :51. `ring` is right to assert — the build was lying about the target. - Silent: `release.yml` builds the shipped macOS arm64 binary on `macos-latest` with this same flag. That runner is `macos-26-arm64`, also `Apple M1 (Virtual)`, and resolves `native` to `neon` on the *pinned* toolchain. Every macOS binary shipped so far was compiled without AES, SHA2, CRC, dotprod or fp16. Nothing errored; it was just quietly slower than intended. `apple-m1` is the correct floor for this target: it is the oldest CPU the target can run on, and it carries the features the platform guarantees. Verified on the runner (aes + sha2 present) and locally (`cargo check -p ring --target aarch64-apple-darwin` builds). Cost on real hardware: on an M4, `native` reports 46 features and `apple-m1` 34. The 12 dropped are bf16, bti, ecv, i8mm, sme, sme2, sme-f64f64, sme-i16i64, v8.5a, v8.6a, v8.7a, wfxt — SME matrix and ML dot-product extensions plus later architecture baselines, none of which the MFT parse or Polars query paths use. aes, sha2, sha3, crc and dotprod are all retained. In exchange, macOS builds become reproducible instead of silently varying with whichever machine ran the build. x86_64-apple-darwin still uses `native` and is untouched: nothing builds or ships on it here, so there is no evidence to act on.
Landed via `just toolchain-sync`, which accepted today's nightly on the first candidate — no step-back needed. rustc 1.99.0-nightly (87e5904f5 2026-07-20). All three lanes the recipe gates on passed before the pin moved: host clippy --all-features -D warnings ✅ host clippy --no-default-features -D warnings ✅ windows cargo xwin clippy x86_64-pc-windows-msvc ✅ linux cargo zigbuild x86_64-unknown-linux-gnu ✅ Re-validated on the new pin afterwards: 2395 tests pass, lint-prod and lint-tests clean, rustdoc clean with --document-private-items --locked. This bump has been deferred since 2026-06-29 on the theory that an upstream `ring 0.17.14` regression blocked it. That theory was wrong — see the preceding commit. The blocker was our own `-C target-cpu=native` on aarch64-apple-darwin, and nothing about the toolchain ever gated it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three threads, all traced from one request: expose the remaining NTFS 3.0+
$STANDARD_INFORMATIONfields at the parse layer.1.
$SIparse layer (the original request)ExtendedStandardInfogainsquota_charged,max_versions,version_number,class_id, populated in the NTFS 3.0+ branch ofparse_standard_info_full. The NTFS 1.2 branch is untouched — that layout has no such fields and..from_attributes(…)correctly leaves them zero.Costs nothing persistent:
ExtendedStandardInfois a stack temporary in both production parsers, and the 56-byteStandardInfothat lands inMftIndexis unchanged. No index, cache, or on-disk format change.StdInfoParse(V30/V12/Malformed/Absent), returned byparse_standard_info_fulland surfaced onParsedRecord::std_info_parse.Previously a
$SIthat could not be decoded returned silently, leaving every timestamp at0— FILETIME 1601-01-01, a value a real record can legitimately hold. Nothing recorded the failure, so a consumer building a forensic record could only report the default as fact.is_corruptdoes not cover this: it is set solely on fixup/USA failure.Measured:
ParsedRecordstays at 256 bytes (the field lands in existing padding),StdInfoParseis 1 byte in a register, and the variant is just which branch the parser already took. Deliberately not#[must_use]— the bulk pipelines have nowhere to record it and dropping it there is correct, so the value is dead and optimised out.Fragment-parser bug fixed.
parse_record_to_fragmenthand-rolled its own$SIread and unconditionally parsed the 36-byte NTFS 1.2 layout regardless of the attribute's real size, silently zeroingusn/security_id/owner_idon every modern volume. Now routed through the shared, fuzz-hardened helper, so all four parsers agree. Latent today (the function is#[deprecated]with no workspace callers) but kept for a planned carving consumer — see Decide whether to remove the deprecated fragment parser (parse_record_to_fragment) #572.Test coverage pins the fields on the public API (
parse_record_full), not just the helper: a refactor givingfull.rsits own$SIread would otherwise keep helper-level tests green while zeroing the fields for every external caller. Verified to fail when that second hop is broken. New pins also cover all threeMalformedconditions, and — the point of the exercise — that a genuine all-zero$SIreturnsV30while an undecodable one returnsMalformedwith byte-identical timestamps.2. macOS builds were compiled as generic ARMv8-A (#553)
-C target-cpu=nativeresolves to baregenericon every GitHub macOS runner. They reportApple M1 (Virtual); LLVM cannot identify the virtualised CPU and falls back to the ARMv8-A baseline. Measured on the runners:nativeyields exactly one feature,neon— identical to-C target-cpu=generic.ringasserts at compile time thataesandsha2are statically present onaarch64-apple-darwin, so it refuses to build. That is the macOS Nightly Canary failure, red weekly since at least 2026-06-08 and diagnosed as an upstreamring 0.17.14regression to wait out. It is neither upstream nor a regression: reproduced locally on the pinned toolchain with-C target-cpu=generic, byte-identical E0080s atdarwin.rs:44and:51.ringis right to assert — the build was lying about the target.ring 0.17.14is still the newest release; there was never an update to wait for.release.ymlbuilds the shipped macOS arm64 binary onmacos-latest(imagemacos-26-arm64) with the same flag, and that runner resolvesnativetoneonon the pinned toolchain. Every macOS binary shipped so far was built without AES, SHA2, CRC, dotprod or fp16. Nothing errored; it was just quietly slower.Fixed by pinning
apple-m1— the oldest CPU the target can run on, carrying the features the platform guarantees. On an M4,nativereports 46 features andapple-m134; the 12 dropped are SME matrix / ML dot-product extensions and later architecture baselines (bf16,bti,ecv,i8mm,sme*,v8.5a–v8.7a,wfxt), none used by the MFT parse or Polars query paths.aes,sha2,sha3,crc,dotprodall retained, and macOS builds stop varying with whichever machine produced them.x86_64-apple-darwinstill usesnativeand is untouched — nothing builds or ships on it here, so there is no evidence to act on.3. Toolchain pin →
nightly-2026-07-21Deferred since 2026-06-29 on the theory that
ringblocked it. That theory was wrong, so the bump is unblocked.just toolchain-syncaccepted today's nightly on the first candidate: host clippy--all-featuresand--no-default-features, Windowscargo xwin clippy, and Linuxcargo zigbuildall green before the pin moved. Re-validated after: 2395 tests,lint-prod,lint-tests, and rustdoc with--document-private-items --locked.Also
fix(docs): twouffs-versionintra-doc links pointed at a#[cfg(feature = "build")]-gated item, failing the pre-push rustdoc gate onmain. Split into its own commit — unrelated cause, independently revertible.docs(mft): theforensic_flagsaccessor comment listed bits 0-2 and stopped, reading as "bit 3 onwards is free". It is not — bit 3 ishas_default_data, which feedscompute_tree_metrics, so a new flag landing there surfaces as wrongtreesize/tree_allocatedfar from the cause. That truncated comment caused a real error in review. Completing it pushedtypes.rspast the 800-LOC policy limit, so the numeric-conversion helpers moved to a newindex/convert.rs(re-exported under identical public paths) rather than trimming the docs or taking a size exception.Verification
2395 tests pass, all pre-push gates green (23 checks incl. Windows xwin clippy), rustdoc clean.
Not yet proven in CI: no run has compiled this repo with
target-cpu=apple-m1. The macOS canary is schedule/dispatch-only, so a green PR does not confirm it. Dispatching the canary against this branch is the run that actually closes that loop.Closes #553.