Repository navigation
Conversation
A new binary that is GNU du 9.x on the command line and in its output, byte for byte, including diagnostics and exit status. It is meant to be linked as du, so whatever already runs du gets it. The walk lists and stats on every core, keeping each directory in the order fts visits it (readdir order, batches over 10,000 entries sorted by inode). A serial pass then applies du's process_file in that order, so first-seen hard links, -x, --exclude, depth and threshold behave as GNU's do. Each operand's walk is kept as a snapshot. By default a later walk reuses a directory's listing when its ctime and mtime are unchanged and older than the snapshot by the racy margin, and still stats every entry, so the answer is exact. --max-age answers from a young snapshot without walking. --json adds kind, reclaim reason, clean command and last write per entry, and says how old the numbers are. tests/gnu.rs compares 73 argument sets against GNU du, cold, with the index, and from it.
getattrlistbulk returns a buffer of entries with their attributes, for about what the listing costs alone, instead of readdir and an fstatat per entry. Only what converts to exactly the struct stat du would have seen is taken from it: regular files and symlinks with one link, their st_blocks being the total allocation rounded up to 512-byte blocks and st_size the data fork's length, as XNU's vn_stat fills them. Directories, hard-linked files, anything else and any entry reported with an error still go through fstatat, and a record that does not parse sends the whole directory back to readdir. On a 6.65M-entry tree this takes a walk from 40 s to 25-29 s (GNU du: 148 s). Output is identical to the stat walk and to GNU du on /Applications, compressed files included, for blocks, apparent sizes and inode counts. DISKTREE_DU_BULK=0 turns it off.
A snapshot is read only when it can help: to answer for --max-age, or where listings can be reused, which with macOS's bulk attributes they cannot. Snapshots are written while the answer prints, not after it. The format stores a device, a readdir inode and a ctime only when they differ from the directory's, the entry's inode and its mtime, and no longer keeps access times; --max-age does not answer --time=atime. A 6.65M-entry snapshot goes from 470 MB to 313 MB, and a walk that writes one takes as long as one that does not.
A snapshot young enough for --max-age is now brought up to date from what changed since it was taken, rather than returned as it was. On macOS the FSEvents journal names every directory whose entries changed or that holds a file written and closed since the snapshot; only those are read again and every other listing is shared from the snapshot. Elsewhere, or when the journal cannot vouch for the whole interval (a new journal UUID, dropped or wrapped events), every directory is stat'ed and read again when its ctime or mtime moved. Neither sees a file still open for writing, since FSEvents reports a write at close, so files last modified within an hour of the snapshot, and files with more than one link, are stat'ed again every time. On a 6.65M-entry tree a catch-up takes 2-3 s through the journal and 3.4 s by directory, against 25 s for a walk. --json reports which one answered. DISKTREE_DU_JOURNAL=0 skips the journal.
Items only macOS uses are gated to it, and on Windows, where the binary only parses its command line, the unused rest is allowed with a reason. Clippy is clean for aarch64-apple-darwin, x86_64-unknown-linux-gnu and x86_64-pc-windows-msvc. The comparison reports every case that differs rather than the first, and knows which GNU it is talking to. Messages are compared with the program name normalised, since coreutils before 9.2 prefixed them with the full argv[0]. It runs against GNU du 9.4 and later (CI's Ubuntu 24.04 has 9.4), skipping two cases GNU changed after that: a negative --max-depth, and the wording of a bad --time-style. It passes on Ubuntu 24.04 (9.4), Debian 13 (9.7) and Homebrew's 9.11. README gains an "As du" section with timings; AGENTS.md no longer says windows.rs holds the workspace's only unsafe code.
The journal's paths are the directory's own. Ask the kernel for the path of the open operand (F_GETPATH) rather than resolving the name it was given, and test that an operand typed in another case than the directory's, on a case-insensitive volume, is still caught up.
- A catch-up by directory now treats a directory whose timestamps were within 2 s of the snapshot's walk as changed, as the default mode's racy-clean rule does. Without it a same-tick change could be carried into a later default run. - --max-age counts from the last full walk, recorded in the snapshot; a catch-up that rewrites the snapshot keeps it, so what a catch-up cannot see is seen within AGE. Snapshots are no longer marked as confirmed. - A journal catch-up reads again every directory on another file system below the operand: its changes are in another journal. - With excludes, the snapshot key includes the operand as spelled, since exclude patterns match that spelling. - Directories deeper than PATH_MAX are opened a piece at a time, as fts does, instead of failing with ENAMETOOLONG. - A directory getattrlistbulk cannot read from the start (readable but not searchable) is listed with readdir, so each entry is reported as GNU does. - The default walk stops at a directory that is its own ancestor (a bind mount), as the report already did. - A bad block size keeps the value gnulib's xstrtol understood: a bad DU_BLOCK_SIZE falls back as GNU's does, and '-B 16Z' is 'too large'. - -X drops trailing white space and blank lines and reads - from stdin; a --files0-from list that fails part way is reported where it fails and the total still printed; an ambiguous option is repeated with its =value. - Standard output failing ends du as GNU's does: status 141 for a closed pipe, 'write error' and 1 otherwise. - Corrupt snapshots cannot recurse without bound or allocate by a bogus count; concurrent writers of one snapshot use distinct temporary files; worker threads get 64 MiB stacks for deep trees. The comparison against GNU gains six cases (a tree deeper than PATH_MAX, a directory that is readable but not searchable, a -X file with CRLF and blank lines, --files0-from naming a directory, an ambiguous option with a value, and one operand spelled two ways with an exclude). README lists the differences that remain.
aronchick
marked this pull request as ready for review
September 29, 2026 15:26
aronchick
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #47
disktree --duis GNU du: the same options, output, messages and exitstatus. It now runs as a headless mode of the existing
disktreebinary,before settings, window or GPUI setup. Everything after
--duis parsed as aGNU du argument. When the binary is invoked through a link named
du, itselects the same mode automatically, so existing scripts and coding agents
can keep calling
duwithout a second binary on disk.Why
Agents run
du -sh *repeatedly over the same tree. Every GNU du run is afull walk, and every answer is a bare number. This makes the walk much cheaper,
makes repeats nearly free when the caller accepts a short catch-up, and can
hand an agent what disktree already knows about a directory: its kind, why it
can go, and the command that frees it.
Numbers
A 670 GB tree of 6.65 million entries, Apple M5 Max, APFS, warm cache:
du -sdisktree --du -sdisktree --du -s --max-age=1h, through the FSEvents journaldisktree --du -s --max-age=1h, by directory (no journal)How it stays exact
crates/disktree-core/src/du/walk.rslists and stats on every core butkeeps each directory's entries in the order GNU's
ftsvisits them:readdirorder, read in batches of 100,000, and a batch over 10,000 entriessorted by inode (skipped on tmpfs, NFS, CIFS and Lustre, as gnulib does).
report.rsportsdu.c'sprocess_fileand applies it in that order, sofirst-seen hard links,
-x,--exclude, depths, thresholds,-Sand--inodescome out as GNU's do.args.rsemulatesgetopt_long: argumentpermutation, clustered short options, unique-prefix long options and their
exact complaints.
num.rsports gnulib'sxstrtolsuffixes andhuman_readableceiling rounding. disktree's own options match only whenspelled in full, so no GNU abbreviation changes meaning.
bulk.rsusesgetattrlistbulkand takes from it only what converts to exactly the
struct statdu wouldhave seen: regular files and symlinks with one link (
st_blocksis totalallocation rounded up to 512-byte blocks, and
st_sizeis data forklength, as XNU's
vn_statfills them). Directories, hard-linked files,anything else and any entry reported with an error go through
fstatat.Output matches the plain walk and GNU on
/Applications, compressed filesincluded.
--max-age, every answer is a full walk. The snapshot iswritten alongside at no measurable cost.
--max-ageand what it can missEach walk is kept as a snapshot (
~/Library/Caches/disktree/duon macOS and~/.cache/disktree/duelsewhere). With--max-age=AGE, a snapshot youngerthan
AGEis brought up to date instead of walked again:the snapshot's walk, names every directory whose entries changed or that
holds a file written and closed since. Only those are read; everything else
is shared from the snapshot. A journal reset (new UUID), dropped or wrapped
events, or a replay that does not finish falls back to the next method.
mtime moved.
A file still open for writing is not in the journal until it closes. Files
modified within an hour of the snapshot, plus every multiply-linked file, are
therefore stat'ed again each time. This covers the usual log or build still
being written. By directory, a file that grew in place is caught only if it
was one of those.
AGEcounts from the last full walk and a catch-up neverresets it, so nothing can go unseen longer than
AGE. A directory on anotherfile system below the operand is always read again in a journal catch-up,
since its changes are in another journal. A unit test covers that path;
mounting a disk image was not available for an end-to-end check.
--freshwalks everything.
--jsonreportsas_ofandsource(walk,journalordirectories) forthe answer. Each entry includes its size, kind, reclaim reason, a clean command
(
cargo clean --manifest-path ...,pnpm store prune,npm cache clean --force, orrm -rffor build output), and last write.Behind a
dusymlink the switches come fromDISKTREE_DU_JSON,DISKTREE_DU_MAX_AGE,DISKTREE_DU_FRESHandDISKTREE_DU_INDEX.Tests
crates/disktree-app/tests/gnu.rsbuilds a fixture with the cases du treatsspecially: hard links within and across operands, symlinks to files,
directories, nothing and a loop, a sparse file, an unreadable directory,
odd names, a 10,050-entry directory, a directory deeper than
PATH_MAX,and one readable but not searchable. It runs 78 argument sets against
the GNU du it finds, three ways each: a fresh walk, with the snapshot, and
caught up with
--max-age. Output, messages and exit status must match.what CI runs) and 9.7 (Debian 13), the last two run in containers. Two
cases GNU changed after 9.4 are compared only against 9.11: a negative
--max-depth(it became signed) and the wording of a bad--time-style.tree, and require the
--max-ageanswer to equal a fresh walk's, throughthe journal and by directory. A unit test replays the journal for a real
write.
the by-directory catch-up, deep paths, other file systems under the operand,
snapshot keys with excludes, non-searchable directories,
xstrtolerrorvalues, write errors,
-Xparsing and corrupt-snapshot bounds. Each isfixed, with a regression case where one could be built.
cargo xtask lintandcargo xtask testpass on macOS. Clippy is clean forx86_64-unknown-linux-gnuandx86_64-pc-windows-msvc.Unsafe
Two modules are macOS-only, in the style of
windows.rs:bulk.rsforgetattrlistbulkandfsevents.rsfor FSEvents through CoreServices. Eachhas a module-level
allowwith a reason and aSAFETYcomment on everyblock. Safe code parses each returned buffer, and a record that does not parse
sends the directory back to
readdir.Relation to #74
Issue #74 proposes
disktree --cli, a readable text tree from the existingbinary. This PR does not implement that tree printer. It establishes the same
early headless dispatch point: select the mode before settings or GPUI, write
the result to stdout, and ship it in the one binary users already have. A
future
--clibranch can sit beside--duat that boundary while using theexisting scan pipeline.
Relation to #32
This does not depend on #32. It uses
disktree-coreclassification directly.If #32 lands, its JSON command surface can call the same
disktree_core::dumodule.Known differences from GNU
The README lists these too. They are all rare:
du >&-exits 0. Rust reopens a closed standard output on/dev/nullbefore
main, so the write error GNU reports never happens.come out in another order because GNU sorts with the unstable
qsort.--timeformats go through chrono, so%Zand%xdiffer. The named stylesmatch.
--excludewildcards match bytes.‘...’.Not done
--max-agestats every directory there.
-B "'1",and
.as the decimal point.st_dev,st_inoandst_blocks, thenexits.