Skip to content

feat(mft): extended $SI fields, parse status, and macOS target-cpu fix#574

Merged
githubrobbi merged 8 commits into
mainfrom
feat/mft-si-status-and-macos-target-cpu
Jul 21, 2026
Merged

feat(mft): extended $SI fields, parse status, and macOS target-cpu fix#574
githubrobbi merged 8 commits into
mainfrom
feat/mft-si-status-and-macos-target-cpu

Conversation

@githubrobbi

Copy link
Copy Markdown
Collaborator

Three threads, all traced from one request: expose the remaining NTFS 3.0+ $STANDARD_INFORMATION fields at the parse layer.

1. $SI parse layer (the original request)

  • ExtendedStandardInfo gains quota_charged, max_versions, version_number, class_id, populated in the NTFS 3.0+ branch of parse_standard_info_full. The NTFS 1.2 branch is untouched — that layout has no such fields and ..from_attributes(…) correctly leaves them zero.
    Costs nothing persistent: ExtendedStandardInfo is a stack temporary in both production parsers, and the 56-byte StandardInfo that lands in MftIndex is unchanged. No index, cache, or on-disk format change.

  • StdInfoParse (V30 / V12 / Malformed / Absent), returned by parse_standard_info_full and surfaced on ParsedRecord::std_info_parse.
    Previously a $SI that could not be decoded returned silently, leaving every timestamp at 0 — FILETIME 1601-01-01, a value a real record can legitimately hold. Nothing recorded the failure, so a consumer building a forensic record could only report the default as fact. is_corrupt does not cover this: it is set solely on fixup/USA failure.
    Measured: ParsedRecord stays at 256 bytes (the field lands in existing padding), StdInfoParse is 1 byte in a register, and the variant is just which branch the parser already took. Deliberately not #[must_use] — the bulk pipelines have nowhere to record it and dropping it there is correct, so the value is dead and optimised out.

  • Fragment-parser bug fixed. parse_record_to_fragment hand-rolled its own $SI read and unconditionally parsed the 36-byte NTFS 1.2 layout regardless of the attribute's real size, silently zeroing usn/security_id/owner_id on every modern volume. Now routed through the shared, fuzz-hardened helper, so all four parsers agree. Latent today (the function is #[deprecated] with no workspace callers) but kept for a planned carving consumer — see Decide whether to remove the deprecated fragment parser (parse_record_to_fragment) #572.

  • Test coverage pins the fields on the public API (parse_record_full), not just the helper: a refactor giving full.rs its own $SI read would otherwise keep helper-level tests green while zeroing the fields for every external caller. Verified to fail when that second hop is broken. New pins also cover all three Malformed conditions, and — the point of the exercise — that a genuine all-zero $SI returns V30 while an undecodable one returns Malformed with byte-identical timestamps.

2. macOS builds were compiled as generic ARMv8-A (#553)

-C target-cpu=native resolves to bare generic on every GitHub macOS runner. They report Apple M1 (Virtual); LLVM cannot identify the virtualised CPU and falls back to the ARMv8-A baseline. Measured on the runners: native yields exactly one feature, neon — identical to -C target-cpu=generic.

  • Loud half: ring asserts at compile time that aes and sha2 are statically present on aarch64-apple-darwin, so it refuses to build. That is the macOS Nightly Canary failure, red weekly since at least 2026-06-08 and diagnosed as an upstream ring 0.17.14 regression to wait out. It is neither upstream nor a regression: reproduced locally on the pinned toolchain with -C target-cpu=generic, byte-identical E0080s at darwin.rs:44 and :51. ring is right to assert — the build was lying about the target. ring 0.17.14 is still the newest release; there was never an update to wait for.
  • Silent half: release.yml builds the shipped macOS arm64 binary on macos-latest (image macos-26-arm64) with the same flag, and that runner resolves native to neon on the pinned toolchain. Every macOS binary shipped so far was built without AES, SHA2, CRC, dotprod or fp16. Nothing errored; it was just quietly slower.

Fixed by pinning apple-m1 — the oldest CPU the target can run on, carrying the features the platform guarantees. On an M4, native reports 46 features and apple-m1 34; the 12 dropped are SME matrix / ML dot-product extensions and later architecture baselines (bf16, bti, ecv, i8mm, sme*, v8.5av8.7a, wfxt), none used by the MFT parse or Polars query paths. aes, sha2, sha3, crc, dotprod all retained, and macOS builds stop varying with whichever machine produced them.

x86_64-apple-darwin still uses native and is untouched — nothing builds or ships on it here, so there is no evidence to act on.

3. Toolchain pin → nightly-2026-07-21

Deferred since 2026-06-29 on the theory that ring blocked it. That theory was wrong, so the bump is unblocked. just toolchain-sync accepted today's nightly on the first candidate: host clippy --all-features and --no-default-features, Windows cargo xwin clippy, and Linux cargo zigbuild all green before the pin moved. Re-validated after: 2395 tests, lint-prod, lint-tests, and rustdoc with --document-private-items --locked.

Also

fix(docs): two uffs-version intra-doc links pointed at a #[cfg(feature = "build")]-gated item, failing the pre-push rustdoc gate on main. Split into its own commit — unrelated cause, independently revertible.

docs(mft): the forensic_flags accessor comment listed bits 0-2 and stopped, reading as "bit 3 onwards is free". It is not — bit 3 is has_default_data, which feeds compute_tree_metrics, so a new flag landing there surfaces as wrong treesize/tree_allocated far from the cause. That truncated comment caused a real error in review. Completing it pushed types.rs past the 800-LOC policy limit, so the numeric-conversion helpers moved to a new index/convert.rs (re-exported under identical public paths) rather than trimming the docs or taking a size exception.

Verification

2395 tests pass, all pre-push gates green (23 checks incl. Windows xwin clippy), rustdoc clean.

Not yet proven in CI: no run has compiled this repo with target-cpu=apple-m1. The macOS canary is schedule/dispatch-only, so a green PR does not confirm it. Dispatching the canary against this branch is the run that actually closes that loop.

Closes #553.

`emit_build_env` is behind `#[cfg(feature = "build")]`, so the two
[`emit_build_env`] links in uffs-version's docs do not resolve when the
crate is documented without that feature. The pre-push rustdoc gate runs
`cargo doc --workspace --no-deps --document-private-items --locked` with
`-D rustdoc::broken-intra-doc-links`, so this failed on main.

Demote both to plain code spans that name the feature gate. The
workspace rustdoc gate now passes.
The NTFS 3.0+ 72-byte $STANDARD_INFORMATION carries four fields past the
timestamps that the parser read into `StandardInformationExtended` and
then dropped: `quota_charged`, `max_versions`, `version_number`, and
`class_id`. Downstream consumers that build a full forensic record had to
re-read them from the raw bytes with their own hard-coded offset table.

Add them to `ExtendedStandardInfo` and populate them in the NTFS 3.0+
branch of `parse_standard_info_full`, the single source of truth for $SI
parsing. The NTFS 1.2 branch is unchanged: that 36-byte layout has no
such fields, and `..from_attributes(...)` correctly leaves them zero.

Costs nothing persistent. `ExtendedStandardInfo` is a stack temporary in
both production parsers, converted immediately via
`StandardInfo::from_extended` and dropped; the 56-byte `StandardInfo`
that lands in `MftIndex` is untouched, so no index, cache, or on-disk
format changes. The values are already materialized by the zerocopy cast,
so this is three u32 moves and one u64.

Additive and semver-minor: every construction site uses `::default()` or
`..from_attributes(...)`, so struct-update syntax covers the new fields.
`parse_record_to_fragment` hand-rolled its own $STANDARD_INFORMATION read
and unconditionally parsed the 36-byte NTFS 1.2 layout, ignoring the
attribute's real value_length. On any NTFS 3.0+ volume — i.e. all of them
— it therefore returned usn, security_id and owner_id as zero while
looking like it had parsed the attribute successfully.

Route it through `parse_standard_info_full`, the same single source of
truth the two live parsers already use, so all three paths agree. The
directory flag is applied after the attribute loop, so moving the read
does not disturb it. Malformed $SI now leaves `std_info` at its default
and continues the loop instead of aborting the whole attribute walk,
matching `unified::process_record`.

The parser is deprecated and has no callers in the workspace, so this is
latent rather than user-visible today; it is kept for a planned carving
consumer, which makes correctness now cheaper than archaeology later.
Also replaces the `too_many_lines` expect reason, which claimed this was
a "performance-critical hot path" — false since the fragment-merge
rewrite, and precisely why the bug went unexamined this long.

Adds a regression pin on the fragment path alongside the existing
two-parser one; verified to fail (usn 0 != 66) against the pre-fix code.
… the helper

The existing pins assert `parse_standard_info_full` directly, but the
public API consumers actually call is `parse_record_full`, which reaches
those fields through two hops: a local `ExtendedStandardInfo` in `full.rs`
that is later moved onto `ParsedRecord::std_info`. A refactor giving
`full.rs` its own $SI read would keep the helper-level tests green while
silently zeroing the fields for every external caller.

Assert quota_charged/max_versions/version_number/class_id — plus the
pre-existing usn/security_id/owner_id, which ride the same two hops — on a
`ParsedRecord` returned by `parse_record_full`. Verified to fail when the
second hop is broken (quota_charged 0 != 65536).

The addition pushed parse/tests.rs to 839 LOC, past the 800-LOC policy
limit. Split rather than granted an exception: tests.rs becomes
tests/mod.rs (698 LOC, helpers and everything else) plus tests/std_info.rs
(152 LOC, all three $STANDARD_INFORMATION pins). Also drops the now-false
`single_call_fn` expectation on `create_file_name_value`, which the new
test makes a second caller of.
….rs out

The comment above the forensic-flag accessors listed only bits 0-2 and
stopped, which reads as "bit 3 onwards is free". It is not: bits 3-5 are
has_default_data, has_i30_stream and is_unified, written by their own
setters, which is why `set_forensic_flags` masks them through with
`& 0b11_1000`. Bit 3 feeds `compute_tree_metrics`, where it separates
"empty $DATA" from "no $DATA", so a new flag landing there would surface
as wrong treesize/tree_allocated far from the cause. Only bits 6-7 are
free. The table is now complete and says so.

types.rs sat at exactly 800 LOC, so the full comment pushed it over the
policy limit. Rather than trim the docs to fit or take a size exception,
lift the "Safe Casting Helpers" section — 15 numeric conversions with no
dependency on the record structs, and no callers inside types.rs — into a
new `index/convert.rs` (211 LOC), leaving types.rs at 610 with room to
document itself. Re-exported from `index.rs` under the same public paths,
so no call site outside the crate changes; `index/base.rs` was the one
in-crate user of a `types::`-qualified path and now points at `convert`.
When $SI could not be decoded, the parser returned silently and left
`ExtendedStandardInfo` at its defaults — every timestamp `0`, i.e. FILETIME
1601-01-01, which is a value a real record can legitimately hold. Nothing
recorded that the attribute had failed, so a consumer building a forensic
record could only report the default as fact. `is_corrupt` does not cover
this: it is set solely by the caller on fixup/USA failure, and there was no
attribute-level damage representation anywhere in the crate.

`parse_standard_info_full` now returns `StdInfoParse`:

  V30       decoded as the 72-byte NTFS 3.0+ layout
  V12       decoded as the 36-byte NTFS 1.2 layout (missing extended
            fields are correct, not damage)
  Malformed present but undecodable — resident header past the end of the
            record, declared length under the 36-byte minimum, or payload
            overrunning the buffer
  Absent    no $SI attribute at all; unusual, but not evidence of damage,
            and deliberately distinct from Malformed

Surfaced on `ParsedRecord::std_info_parse` so it reaches the public API
(`parse_record_full`, `parse_record_forensic`), which is where an external
index reads from. Deliberately not `#[must_use]`: the bulk pipelines
(unified, direct-index, fragment) have nowhere to record it, and dropping
it there is the correct choice, not an oversight worth a lint.

Costs nothing measurable. `ParsedRecord` stays at 256 bytes — the field
lands in existing padding — and `StdInfoParse` is 1 byte returned in a
register. The variant is just which branch the parser already took, so no
new work is done; the three bulk callers discard it and the value is dead.
No index, cache, or on-disk format change.

Groundwork only: representation (columns, DataFrame projection, the
daemon's shmem row) is deliberately left to the consumer, which builds its
own index. Closes #573.
`-C target-cpu=native` resolves to bare `generic` on every GitHub macOS
runner: they report `Apple M1 (Virtual)`, and LLVM's host detection
cannot identify the virtualised CPU, so it falls back to the ARMv8-A
baseline. Measured on the runners themselves — `native` yields exactly
one feature, `neon`, identical to `-C target-cpu=generic`.

Two consequences, one loud and one silent:

- Loud: `ring` asserts at compile time that `aes` and `sha2` are
  statically present on aarch64-apple-darwin, so it refuses to build.
  This is the macOS Nightly Canary failure (#553), red weekly since at
  least 2026-06-08. It was diagnosed as an upstream `ring 0.17.14`
  regression to wait out; it is neither upstream nor a regression.
  Reproduced locally on the pinned toolchain with
  `-C target-cpu=generic`: byte-identical E0080s at darwin.rs:44 and :51.
  `ring` is right to assert — the build was lying about the target.
- Silent: `release.yml` builds the shipped macOS arm64 binary on
  `macos-latest` with this same flag. That runner is `macos-26-arm64`,
  also `Apple M1 (Virtual)`, and resolves `native` to `neon` on the
  *pinned* toolchain. Every macOS binary shipped so far was compiled
  without AES, SHA2, CRC, dotprod or fp16. Nothing errored; it was just
  quietly slower than intended.

`apple-m1` is the correct floor for this target: it is the oldest CPU
the target can run on, and it carries the features the platform
guarantees. Verified on the runner (aes + sha2 present) and locally
(`cargo check -p ring --target aarch64-apple-darwin` builds).

Cost on real hardware: on an M4, `native` reports 46 features and
`apple-m1` 34. The 12 dropped are bf16, bti, ecv, i8mm, sme, sme2,
sme-f64f64, sme-i16i64, v8.5a, v8.6a, v8.7a, wfxt — SME matrix and
ML dot-product extensions plus later architecture baselines, none of
which the MFT parse or Polars query paths use. aes, sha2, sha3, crc and
dotprod are all retained. In exchange, macOS builds become reproducible
instead of silently varying with whichever machine ran the build.

x86_64-apple-darwin still uses `native` and is untouched: nothing builds
or ships on it here, so there is no evidence to act on.
Landed via `just toolchain-sync`, which accepted today's nightly on the
first candidate — no step-back needed. rustc 1.99.0-nightly (87e5904f5
2026-07-20).

All three lanes the recipe gates on passed before the pin moved:

  host    clippy --all-features -D warnings          ✅
  host    clippy --no-default-features -D warnings   ✅
  windows cargo xwin clippy x86_64-pc-windows-msvc   ✅
  linux   cargo zigbuild x86_64-unknown-linux-gnu    ✅

Re-validated on the new pin afterwards: 2395 tests pass, lint-prod and
lint-tests clean, rustdoc clean with --document-private-items --locked.

This bump has been deferred since 2026-06-29 on the theory that an
upstream `ring 0.17.14` regression blocked it. That theory was wrong —
see the preceding commit. The blocker was our own
`-C target-cpu=native` on aarch64-apple-darwin, and nothing about the
toolchain ever gated it.
@githubrobbi
githubrobbi added this pull request to the merge queue Jul 21, 2026
Merged via the queue into main with commit 73cf363 Jul 21, 2026
24 checks passed
@githubrobbi
githubrobbi deleted the feat/mft-si-status-and-macos-target-cpu branch July 21, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐤 Nightly Canary: upcoming-toolchain regression — d0402d5

1 participant