refactor fastcp/fastmd5's direct_io and file system walking code to a shared location so they can be used by other utilities.

CLAUDE.md

@@ -0,0 +1,125 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+`mlperf-common` is a grab-bag of utilities shared across NVIDIA's MLPerf benchmark
+submissions. Benchmarks consume it by adding
+`git+https://github.com/NVIDIA/mlperf-common.git` to their `requirements.txt`.
+
+Two mostly independent halves live here:
+
+* **Logging / profiling** (`mlperf_common/logging.py`, `scaleoutbridge.py`,
+  `frameworks/`, `callbacks/`) — wraps the official `mlperf_logging` package.
+* **Fast file I/O** (`mlperf_common/fileio/`, `client/`, `src/`) — O_DIRECT copy
+  and checksum tools, and the shared tree-walk and I/O primitives behind them.
+
+## Commands
+
+```bash
+python3 tests/run_tests.py          # whole suite (stdlib only, no pytest, no GPU)
+python3 tests/test_copyplan.py      # one test file, directly
+
+make -C src                         # build the C++ tools
+make -C src install prefix=/usr/local
+
+pip install .                       # installs the package + client/ scripts into bin/
+```
+
+There is no linter or formatter configured, and no CI in the repo.
+
+Tests run each file in a **separate interpreter** so nothing one leaves in
+`sys.modules` reaches another. They are stdlib-only and need no GPU. See
+`tests/README.md` for what they do and do not cover — notably not `direct_io`
+and not O_DIRECT itself.
+
+## Architecture
+
+### Logging stack
+
+`MLLoggerWrapper` (`mlperf_common/logging.py`) is the entry point. It takes a
+`CommunicationHandler` so the same wrapper works under either `torch.distributed`
+or MPI; `frameworks/base.py` defines the `CommunicationHandler` /
+`ProfilerHandler` interfaces and `frameworks/{pyt,mxnet,hugectr,base_mpi}.py`
+implement them per framework. Adding framework support means implementing those
+two interfaces, not touching the wrapper.
+
+`scaleoutbridge.py` layers profiling on top: `init_bridge(prof_handler,
+comm_handler, mllogger)` picks a bridge implementation from env vars —
+`TIME_TAGS` / `NVTX_FLAG` select `ScaleoutBridgeIterwise`, `EPOCH_PROF` selects
+`ScaleoutBridgeEpochwise`, and with none set you get the no-op
+`ScaleoutBridgeBase`.
+
+`callbacks/logging.py` is the Lightning/NeMo layer (`LoggingCallback`,
+`MLPerfLogger`, `StatsLogCallback`). It imports `lightning.pytorch` with a
+fallback to `pytorch_lightning`, and is knob-driven by env vars
+(`FORCE_SUCCESS_STATUS`, `REDUCE_TP`, `LOG_EVERY_N_BATCHES`, `RUN_N_ITERS`,
+`SEED`). This is the only part that pulls in Lightning and numpy.
+
+### fileio
+
+Two modules, with a deliberate dependency rule stated in
+`mlperf_common/fileio/__init__.py`: **neither may import torch.** They stay
+stdlib-only so the single-node `client/` scripts don't drag in a training stack.
+Anything needing torch belongs in a module of its own.
+
+* `direct_io.py` — aligned-buffer `pread`/`pwrite` with retry loops.
+  `allocate_aligned_buffers` carves per-thread buffers out of one big ctypes
+  allocation and hands back **memoryviews**, so slicing a buffer to the length
+  actually read costs nothing.
+* `copyplan.py` — source-tree walk and src→dst mapping. `plan_copy_operations`
+  raises `UnreadableEntries` listing *every* bad entry rather than dying on the
+  first, because a walk that silently skips an unlistable subtree yields a
+  partial copy that exits 0 — and then a checksum run that skips the same files
+  and agrees with it.
+
+Both tools align to 2 MiB, the Linux huge-page size. It is *not* a shared
+constant -- it is hardcoded in each (`fastcp:90,180`, `fastmd5:44,121`). Worth
+promoting into `direct_io` next to `round_up`, so the two cannot drift.
+
+This package was extracted from `fastcp` and `fastmd5`, which had been carrying
+their own copies. Keep the two scripts going through it rather than reintroducing
+private variants.
+
+### client/ and src/
+
+`client/` holds the single-node scripts. `setup.py`'s `scripts=` installs
+`bindpcie` (NUMA/IB affinity binding), `mgpurun`, `slurm2pytorch` (derives
+PyTorch rendezvous env from SLURM), `fastcp`, `direct_io.py`, and the
+log/telemetry shell helpers.
+
+**`fastmd5` and `dropcache` are in `client/` but are NOT in `scripts=`**, so a
+`pip install` does not produce them -- the only supported way to run `fastmd5`
+is in place from a checkout. That matters more now that it hard-depends on the
+package, because its own ImportError advice says "install mlperf-common", a
+remedy that never yields a `fastmd5`. Either add it to `scripts=` or fix the
+message; don't leave both.
+
+`fastcp` and `fastmd5` are a threaded copy and a per-GB checksum. `fastcp`
+opens with O_DIRECT; `fastmd5` does not, despite using the same aligned-buffer
+machinery.
+
+**Don't delete `slurm2pytorch`.** Benchmarks outside this repo depend on it and
+it stays installed, even though nothing in this repo calls it.
+
+These scripts import `direct_io` and `mlperf_common.fileio.copyplan` via a
+`sys.path` dance that works both for a pip install and for a source tree with
+`mlperf_common/` alongside `client/` (the "deploy the repo to a shared filesystem
+and run in place" pattern). `client/direct_io.py` is a compatibility shim
+re-exporting `mlperf_common.fileio.direct_io`. Copying individual scripts out of
+`client/` on their own does not work — preserve that when touching imports, and
+if you add a script remember to add it to `scripts=` in `setup.py`.
+
+`src/` is a separate, older C++ implementation (`fastcp`, `cp-into`,
+`alloc-empty-file-buffer`) sharing `cmdline.h`. It is not built or installed by
+`setup.py`; use the `Makefile` directly.
+
+## Notes
+
+* Commit subjects follow `area: lowercase imperative summary`.
+* New files carry the Apache 2.0 header with an NVIDIA copyright line.
+* The README's "Mount check" section documents `get-mount-info.sh` /
+  `verify-mounts.sh`, which no longer exist — `mlperf_common/mountcheck.py`
+  (`--initialize` / verify against `expected-mounts.csv`, sparse SHA256
+  fingerprints) replaced them. Trust the module over the README there.

FASTCP-BUGS.md

@@ -0,0 +1,82 @@
+# Known defects in fastcp / fastmd5 / fileio
+
+Found reviewing the `mlperf_common.fileio` extraction (PR #43).
+
+Everything here was **introduced by that extraction** — each item was checked
+against `main`'s pre-refactor `client/fastcp` and does not occur there. The much
+longer list of defects this code inherited from `main` is deliberately *not*
+here: those predate the refactor and are not this branch's to fix. They are
+recorded in `MAIN-BUGS.md`.
+
+`direct_io.py` is byte-identical to `main`'s modulo comments, so nothing in it
+is a new defect.
+
+---
+
+## 1. The "cannot import mlperf_common" advisory is unreachable
+
+`client/fastcp:35-42` and `client/fastmd5:20-27` wrap the `copyplan` import in a
+`try`/`except ImportError` that exits with a one-line explanation naming both
+remedies. It never runs: the *unguarded* `import direct_io` at `fastcp:32` /
+`fastmd5:18` executes first and fails first.
+
+Reproduced — copy `client/fastcp` and `client/direct_io.py` somewhere with no
+`mlperf_common` and run it:
+
+```
+Traceback (most recent call last):
+  File "./fastcp", line 32, in <module>
+    import direct_io
+  ...
+ImportError: No module named 'mlperf_common'. direct_io lives in the
+mlperf_common package. ...
+```
+
+A chained three-frame traceback, in the exact scenario the friendly message was
+written for. The shim's own text is reasonable, so this is cosmetic — but the
+handler above it is dead code and reads as though it works.
+
+Fix: move `import direct_io` inside the same guarded block, or drop the
+now-redundant handler.
+
+## 2. A stale `direct_io.py` beside the script silently shadows the package
+
+`sys.path.insert(0, _HERE)` puts the script's own directory ahead of everything,
+so a leftover standalone `direct_io.py` in an install's `bin/` outranks the
+shim. Reproduced with `main`'s pre-refactor copy dropped next to `fastcp`:
+
+```
+direct_io resolved to : .../bin/direct_io.py
+same module object    : False
+same pread function   : False
+```
+
+The copy then runs the *stale* I/O primitives while `copyplan` comes from the
+package. Harmless today only because the two are byte-identical — which means
+the first edit to `mlperf_common/fileio/direct_io.py` silently stops reaching
+any install carrying a leftover `bin/direct_io.py`, with no warning.
+
+This is a live risk precisely because `setup.py` *does* install
+`client/direct_io.py` into `bin/`, so upgrading from a pre-refactor install
+leaves the old file there.
+
+Related, not reproduced: `os.path.abspath` does not resolve symlinks, so
+`ln -s <repo>/client/fastcp ~/bin/fastcp` makes `_HERE` `~/bin` and both inserts
+point at the wrong tree. `os.path.realpath` would fix both.
+
+## 3. `test_copyplan.py`'s refusal rows do not check *which* error was raised
+
+`tests/test_copyplan.py:106-110` asserts only that some `CopyArgumentError`
+escaped:
+
+```python
+try:
+    copyplan.validate_copy_args(sources, destination, **kwargs)
+    check(name, False, "no exception raised")
+except copyplan.CopyArgumentError:
+    check(name, True)
+```
+
+A misspelled fixture path, or a validator that grew over-eager and rejects the
+wrong thing for the wrong reason, leaves every one of these rows green. Match
+the message, or at least a distinct exception subclass per rule.