From 51093507625c91a69c2722f7e442e50a60d1b546 Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 15:12:50 +0100 Subject: [PATCH 01/15] infra: the NUL-framed sibling of readByteSafeLine, for `git log -z` streams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The git-porcelain-framing work (specs/git-porcelain-framing, audit finding #4) re-frames every `git log --name-only` consumer onto NUL-delimited fields, where an EMPTY field is the record boundary. The byte reader that needs is the sibling of readByteSafeLine, not the line reader itself: -z terminates every --format field and every path with NUL, so a zero-length field is a SUCCESSFUL read here, and only end-of-stream with nothing read returns false. Same parity contract as the line reader (one fgetc per byte, EOF as an int sentinel distinct from every byte value, the only narrowing explicit through `unsigned char`). Kept a separate function rather than a delimiter parameter on the line reader because the two delimiters answer different questions — "where does this line end" vs "is this field the record boundary" — and a caller that confused them would read a hostile filename as commit metadata. The line reader stays exactly as it is; its stdio-request-cap deliberations are unaffected. No caller yet: the shared parser that uses this lands next. --- src/infra/stdinline.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/src/infra/stdinline.h b/src/infra/stdinline.h index 214760293..e6ba49f87 100644 --- a/src/infra/stdinline.h +++ b/src/infra/stdinline.h @@ -136,4 +136,31 @@ inline bool readByteSafeLineBounded( std::FILE* in, std::string& line, std::size return detail::readByteSafeLineCore( in, line, maxBytes, overflowed ); } +// The NUL-framed sibling of readByteSafeLine, for `git log -z` byte streams: -z terminates every +// --format field and every --name-only path with NUL instead of '\n', and an EMPTY field is the record +// boundary the parser must see — so, unlike the line reader, a zero-length field is a SUCCESSFUL read +// here, and only end-of-stream with nothing read returns false. Same parity contract as +// readByteSafeLine: one fgetc per byte, EOF as an int sentinel distinct from every byte value, the only +// narrowing explicit through `unsigned char`. Kept a separate function rather than a delimiter +// parameter on the line reader because the two delimiters answer different questions ("where does this +// line end" vs "is this field the record boundary") and a caller that confused them would read a +// hostile filename as commit metadata. +inline bool readNulSafeField( std::FILE* in, std::string& field ) +{ + ASSUME( in != nullptr ); + + field.clear(); + bool didReadAnyByte = false; + for( int byteOrEof = std::fgetc( in ); byteOrEof != EOF; byteOrEof = std::fgetc( in ) ) + { + didReadAnyByte = true; + if( byteOrEof == '\0' ) + { + return true; // delimiter consumed, not appended + } + field.push_back( static_cast( static_cast( byteOrEof ) ) ); + } + return didReadAnyByte; // EOF: a final unterminated field still counts; nothing read is the end +} + } // namespace rw From 5ea40f8aea878cd488d89c00b7282416b1064890 Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 15:41:53 +0100 Subject: [PATCH 02/15] gitmine: the shared NUL-framed record reader, and all five consumers on it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit finding #4. Every `git log --name-only` consumer in this file framed its stream with a LINE-based reader and a `__C__` marker matched as a line PREFIX — three spellings across five sites — so a path whose name begins with the marker bytes was read as commit metadata. Measured on the hostile fixture (a repo whose files include `__C__`, `__C__x|1694000000`, `__C__ 1694000000`, `ab.cpp`, `q"uote.cpp`, `sp ace.cpp`, `back\slash.cpp`, `pi|pe.cpp`): the file `__C__x|1694000000` was read as a HEADER carrying author `x`, and every path after it in byte order was attributed to that author — five real files owned by an email that appears in no commit. The exact-marker file `__C__` was read as a record boundary and the record after it lost its attribution; the forged email itself leaked out as a phantom path row. A C-quoted `a\nb.cpp` never resolved to its real path at all. One reader now parses the stream for all five consumers: `git log -z --name-only --format=%x00__C__%x00%ae%x00%at%x00%ct`. Fields and paths are NUL-terminated and RAW (core.quotepath=false: no C-quoting, so any POSIX-legal filename round-trips byte-exact), and the marker is a WHOLE-FIELD sentinel a path can never spell. The five consumers — gitLogFileSets, gitLogNameOnlyRaw, gitFileCommitCountsInDayWindow, gitLogDecayedFileMining, gitFileAuthors — become visitors over forEachGitLogRecord; their three command shapes collapse into the one gitLogRecordsCmd, which is also where train 28's --no-ext-diff --no-textconv hardening now rides, once, instead of on five lines that could drift apart. The K51 walk-status contract (unread/notRun/exitCode) is carried on the shared GitLogWalk and applied by the reader itself, so it survives the migration unchanged. The layout this parser reads is pinned in specs/git-porcelain-framing/evidence/layout-pin.md (derived empirically, not from git's documentation: the '\n' before a commit's first path and the end-of-stream unterminated field are implementation details, and test/gitframingcheck.sh re-pins them with a raw-stream digest). markWalkUnread/noteWalkExit lose their last caller here; they are removed with the rest of the dead code in a follow-up commit. --- src/gitmine.h | 474 +++++++++++++++++++++++------------------ test/churnjoincheck.sh | 14 +- 2 files changed, 278 insertions(+), 210 deletions(-) diff --git a/src/gitmine.h b/src/gitmine.h index 3297a6bc9..d6f076c5d 100644 --- a/src/gitmine.h +++ b/src/gitmine.h @@ -735,6 +735,182 @@ inline GitCommandLines gitCommandLines( const std::string& cmd ) return out; } +// ── the ONE git-log record frame (audit finding #4): every --name-only consumer reads through here ── +// +// The old frame was a LINE frame: `--format=tformat:__C__…` printed a marker LINE, and the five +// consumers below each re-derived it with their own test (exact match, rfind prefix, 5-byte prefix). A +// line marker is ambiguous three ways at once — a committed file named `__C__` IS the marker line, a +// file named `__C__x|` prefix-matches it, and a filename containing '\n' splits into phantom +// records — and the ownership walk's `__C__%ae|%at` header let one forged name attribute every path +// after it to a forged author. `git log -z` closes all three: fields and paths are NUL-terminated and +// RAW (no C-quoting), and the marker is a WHOLE-FIELD sentinel a path can never spell. +// +// The pinned byte layout (specs/git-porcelain-framing/evidence/layout-pin.md; the prototype command +// that produced it is in the spec's Implementation Notes): +// +// \0 __C__ \0 \0 \0 [ \0 ] [ \n | ] \0 \0 … +// +// Every record opens with an empty field, then the marker, then its three metadata fields. A commit +// with paths follows the metadata with a '\n'-prefixed first path (git's separator); a path-less +// commit stops after its metadata. The next record's opening empty field is the record boundary — +// and an empty field can only come from the format, because paths are never empty. A field that ends +// at end-of-stream without its delimiter still counts; readNulSafeField returns it. +struct GitLogRecord +{ + std::string authorEmail; + std::int64_t authorTs = 0; + std::int64_t committerTs = 0; + std::vector paths; +}; + +// The walk's exit status, carried so the K51 contract (a prefix is not an answer) survives the +// migration: `unread` — git would not start, or died part-way, so the records are empty or a prefix; +// `notRun` — git never ran at all (no popen, not on PATH, not executable), so there is no prefix either. +struct GitLogWalk +{ + std::vector records; + bool unread = false; + bool notRun = false; + int exitCode = 0; +}; + +// The NUL-framed field iterator over one `git log -z` stream: every --format field and every +// --name-only path arrives as one readNulSafeField call, and a zero-length field — the record +// boundary — is a successful read, not the end of the stream. +struct NulFieldStream +{ + std::FILE* pipe = nullptr; + + bool next( std::string& field ) { return readNulSafeField( pipe, field ); } +}; + +// The marker as a whole field. A path may carry these bytes; only a field that IS this string opens a record. +inline constexpr const char* kGitLogMarker = "__C__"; + +// An epoch field: git's decimal seconds, 0 when the field is empty or does not parse. +inline std::int64_t parseEpochField( const std::string& field ) noexcept +{ + if( field.empty() ) { return 0; } + char* end = nullptr; + const std::int64_t v = std::strtoll( field.c_str(), &end, 10 ); + return ( end != field.c_str() ) ? v : 0; +} + +// A pclose()/wait status as an exit code: WEXITSTATUS on a normal exit, -1 for a signal or an unreadable status. +inline int decodedExitCode( int status ) noexcept +{ + return WIFEXITED( status ) ? WEXITSTATUS( status ) : -1; +} + +// The one command every --name-only consumer shares. `extraArgs` is the per-consumer tail: the window +// args (`--since=…`) and, for the single-file ownership query, the trailing `-- ` pathspec. +// The diff-program hardening (--no-ext-diff --no-textconv, upstream train 28) and the +// combined-diff flag ride HERE, once, instead of on five command lines that could drift apart. +// +// extraArgs rides at the END, after every option, and that position is LOAD-BEARING: a pathspec is only +// a pathspec when it comes last, and everything after `--` is read as a path or revision. Put the +// pathspec first and git consumes the `-z --name-only --format=…` that follows it as path arguments — +// the flags never apply, git answers in its DEFAULT format, and the reader sees a stream with no marker +// field in it. Measured: `--owners=SYM` answered "git unavailable / no history" on a healthy one-commit +// repo while the all-files form beside it was correct. A `--since=` window is an option and rides +// anywhere; both forms end in a space, which is what keeps the redirect from gluing on. +inline std::string gitLogRecordsCmd( const std::string& root, const std::string& extraArgs ) +{ + return gitCmd( " -c core.quotepath=false -C " ) + shSingleQuote( root ) + + " log --no-ext-diff --no-textconv " + kMergeDiffArgs + + "-z --name-only --format=%x00__C__%x00%ae%x00%at%x00%ct " + extraArgs + "2>/dev/null"; +} + +// Stream every record of one `git log --name-only` walk to `visit`, one record in memory at a time, so +// the whole-history walk costs what it always cost. Returns the walk's exit status; the records arrive +// through the visitor in the stream's order (newest commit first, git's order). +template +inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::string& extraArgs, Visitor&& visit ) +{ + GitLogWalk walk; + const std::string cmd = gitLogRecordsCmd( root, extraArgs ); + std::FILE* pipe = os::popen( cmd.c_str(), "r" ); + if( !pipe ) + { + walk.unread = true; + walk.notRun = true; + walk.exitCode = -1; + return walk; + } + + NulFieldStream fields{ pipe }; + std::string field; + + GitLogRecord cur; + bool inRecord = false; // a record is open (its opening empty field was seen) + bool sawMarker = false; // ...and its marker was read: a record without one is never visited + bool wantMarker = false; // the next field must be the marker + int metaLeft = 0; // metadata fields still to read for this record + bool havePath = false; // this record has taken its first path + + const auto closeRecord = [ & ]() + { + if( inRecord && sawMarker ) { visit( cur ); } + cur = GitLogRecord{}; + sawMarker = false; + }; + + while( fields.next( field ) ) + { + if( gitWalksAbandoned() ) { break; } // the walk readers' abandon contract: stop at the next field; pclose ends the child + if( field.empty() ) + { + closeRecord(); + inRecord = true; + wantMarker = true; + metaLeft = 3; + havePath = false; + continue; + } + if( wantMarker ) + { + wantMarker = false; + if( field == kGitLogMarker ) { sawMarker = true; } // anything else: a malformed stream — skip to the next boundary + continue; + } + if( metaLeft > 0 ) + { + if( metaLeft == 3 ) { cur.authorEmail = field; } + else if( metaLeft == 2 ) { cur.authorTs = parseEpochField( field ); } + else { cur.committerTs = parseEpochField( field ); } + --metaLeft; + continue; + } + if( !havePath ) + { + havePath = true; + if( field.front() == '\n' ) { field.erase( 0, 1 ); } // git's separator before a commit's first path + } + cur.paths.push_back( std::move( field ) ); + } + closeRecord(); + + walk.exitCode = decodedExitCode( os::pclose( pipe ) ); + if( walk.exitCode != 0 ) + { + walk.unread = true; + walk.notRun = ( walk.exitCode == 127 || walk.exitCode == 126 ); + } + return walk; +} + +// The collecting form, for the one consumer that keeps every record: the raw stream the churn / +// co-change core is resolved against, memoized per (repo, HEAD, window). +inline GitLogWalk gitLogRecords( const std::string& root, const std::string& extraArgs ) +{ + GitLogWalk walk; + const GitLogWalk streamed = forEachGitLogRecord( root, extraArgs, [ & ]( const GitLogRecord& r ) { walk.records.push_back( r ); } ); + walk.unread = streamed.unread; + walk.notRun = streamed.notRun; + walk.exitCode = streamed.exitCode; + return walk; +} + // The repo TOPLEVEL enclosing `absDir` (absolute and already realpath'd), or empty when `absDir` is not inside // a git repo at all. realpath'd on the way out so it compares byte-for-byte with a realpath'd file path — on // this platform `/tmp` is a symlink to `/private/tmp`, and an unresolved toplevel would fail every prefix test @@ -1501,35 +1677,20 @@ inline std::vector> gitLogFileSets( const std::string // open --name-only pipe would leave a filled pipe buffer waiting on us while we wait on the probe. const GitPathIndex byGitPath = gitPathIndexOfFiles( ing, onlyRoot ); - const std::string cmd = gitCmd( " -c core.quotepath=false -C " ) + shSingleQuote( root ) + " log --no-ext-diff --no-textconv " + kMergeDiffArgs + windowArgs + "--name-only --format=tformat:__C__ 2>/dev/null"; - std::FILE* pipe = os::popen( cmd.c_str(), "r" ); - if( !pipe ) - { - return sets; - } - std::vector cur; const auto flush = [ & ]() { recordCommitFileSet( cur, maxFiles, sets, outCensus ); }; - std::string s; - while( readByteSafeLine( pipe, s ) ) // F6: THE line reader, not a char[4096] a long path can be split across + forEachGitLogRecord( root, windowArgs, [ & ]( const GitLogRecord& rec ) { - while( !s.empty() && ( s.back() == '\n' || s.back() == '\r' ) ) - { - s.pop_back(); - } - if( s == "__C__" ) { flush(); continue; } - if( s.empty() ) - { - continue; - } - const std::uint32_t f = resolveGitPath( byGitPath, s ); - if( f != UINT32_MAX ) + for( const std::string& p : rec.paths ) { - cur.push_back( f ); + const std::uint32_t f = resolveGitPath( byGitPath, p ); + if( f != UINT32_MAX ) + { + cur.push_back( f ); + } } - } - flush(); - os::pclose( pipe ); + flush(); + } ); return sets; } @@ -1604,12 +1765,6 @@ inline void markWalkUnread( RawCommitStream& out, int exitCode, bool neverStarte out.notRun = neverStarted || exitCode == 127 || exitCode == 126; } -// A pclose()/wait status as an exit code: WEXITSTATUS on a normal exit, -1 for a signal or an unreadable status. -inline int decodedExitCode( int status ) noexcept -{ - return WIFEXITED( status ) ? WEXITSTATUS( status ) : -1; -} - // K51: fold the walk's pclose() status into `out` — 0 is a clean read and changes nothing. inline void noteWalkExit( RawCommitStream& out, int status ) noexcept { @@ -1623,42 +1778,17 @@ inline RawCommitStream gitLogNameOnlyRaw( const std::string& root, const std::st { PROFILE_SCOPE_DESCRIBE( "gitmine: gitLogNameOnlyRaw (git log --name-only popen)" ); RawCommitStream out; - const std::string cmd = gitCmd( " -c core.quotepath=false -C " ) + shSingleQuote( root ) - + " log --no-ext-diff --no-textconv " + kMergeDiffArgs + "--since=" + shSingleQuote( defaultWindowSince( root, coSince ) ) // F1: HEAD-anchored when `coSince` is a default month window - + " --name-only --format=tformat:__C__%x20%ct 2>/dev/null"; - std::FILE* pipe = os::popen( cmd.c_str(), "r" ); - if( !pipe ) - { - markWalkUnread( out, -1, true ); - return out; - } - - std::string s; - while( readWalkLine( pipe, s ) ) // F6: THE line reader, not a char[4096] a long path can be split across; stops when abandoned - { - while( !s.empty() && ( s.back() == '\n' || s.back() == '\r' ) ) - { - s.pop_back(); - } - if( s.rfind( "__C__", 0 ) == 0 ) // new commit marker: "__C__ " - { - const std::int64_t epoch = ( s.size() > 6 ) ? std::strtoll( s.c_str() + 6, nullptr, 10 ) : 0; - out.commits.push_back( RawCommitStream::Commit{ epoch, {} } ); - continue; - } - if( s.empty() ) - { - continue; - } - if( out.commits.empty() ) - { - out.commits.push_back( RawCommitStream::Commit {} ); // defensive: tformat always emits the marker first, but never crash on a malformed stream - } - out.commits.back().paths.push_back( std::move( s ) ); // readByteSafeLine clear()s its buffer first, so moving out of it is safe - } - // K51: git died (a missing or corrupt object, a bad ref) — the lines above are a prefix, not the window — or the shell - // never ran it (127: not on PATH, 126: not executable), and there are no lines at all. - noteWalkExit( out, os::pclose( pipe ) ); + // F1: HEAD-anchored when `coSince` is a default month window (defaultWindowSince resolves it) + const std::string extraArgs = "--since=" + shSingleQuote( defaultWindowSince( root, coSince ) ) + " "; + const GitLogWalk walk = forEachGitLogRecord( root, extraArgs, [ & ]( const GitLogRecord& rec ) + { + out.commits.push_back( RawCommitStream::Commit{ rec.committerTs, rec.paths } ); + } ); + // K51: git died (a missing or corrupt object, a bad ref) — the records above are a prefix, not the + // window — or the shell never ran it (127: not on PATH, 126: not executable), and there are none. + out.unread = walk.unread; + out.notRun = walk.notRun; + out.exitCode = walk.exitCode; return out; } @@ -1839,57 +1969,32 @@ inline std::vector gitFileCommitCountsInDayWindow( const std::str const GitPathIndex byGitPath = gitPathIndexOfFiles( ing ); const auto resolve = [ & ]( const std::string& gp ) { return resolveGitPath( byGitPath, gp ); }; - // Stream `git log --name-only` with each commit's epoch on its __C__ marker line: "__C__ ". - // A commit counts once per file it touches, iff its epoch is within [cutoff, headEpoch]. - const std::string cmd = gitCmd( " -c core.quotepath=false -C " ) + shSingleQuote( root ) - + " log --no-ext-diff --no-textconv " + kMergeDiffArgs + "--name-only --format=tformat:__C__%x20%ct 2>/dev/null"; - std::FILE* pipe = os::popen( cmd.c_str(), "r" ); - if( !pipe ) + // Stream `git log --name-only` through the shared NUL-framed reader: one record per commit, its + // committer epoch in hand. A commit counts once per file it touches, iff its epoch is within + // [cutoff, headEpoch]. + std::vector cur; // this commit's resolved fileIds (dedup before tally) + forEachGitLogRecord( root, "", [ & ]( const GitLogRecord& rec ) { - return counts; - } - - bool inWindow = false; // is the commit we are currently reading files for in-window? - std::vector cur; // this commit's resolved fileIds (dedup before tally) - const auto flush = [ & ]() - { - if( inWindow ) + if( rec.committerTs < cutoff || rec.committerTs > headEpoch ) { - std::sort( cur.begin(), cur.end() ); - cur.erase( std::unique( cur.begin(), cur.end() ), cur.end() ); - for( std::uint32_t f : cur ) - { - ++counts[f]; - } + return; // out of window: this commit's files do not count } - cur.clear(); - }; - std::string s; - while( readByteSafeLine( pipe, s ) ) // F6: THE line reader, not a char[4096] a long path can be split across - { - while( !s.empty() && ( s.back() == '\n' || s.back() == '\r' ) ) - { - s.pop_back(); - } - if( s.rfind( "__C__", 0 ) == 0 ) // new commit marker: "__C__ " + for( const std::string& gp : rec.paths ) { - flush(); - const std::int64_t epoch = ( s.size() > 6 ) ? std::strtoll( s.c_str() + 6, nullptr, 10 ) : 0; - inWindow = ( epoch >= cutoff && epoch <= headEpoch ); - continue; - } - if( s.empty() || !inWindow ) - { - continue; + const std::uint32_t f = resolve( gp ); + if( f != UINT32_MAX ) + { + cur.push_back( f ); + } } - const std::uint32_t f = resolve( s ); - if( f != UINT32_MAX ) + std::sort( cur.begin(), cur.end() ); + cur.erase( std::unique( cur.begin(), cur.end() ), cur.end() ); + for( std::uint32_t f : cur ) { - cur.push_back( f ); + ++counts[f]; } - } - flush(); - os::pclose( pipe ); + cur.clear(); + } ); return counts; } @@ -2070,20 +2175,25 @@ inline DecayedChurnMined gitLogDecayedFileMining( const std::string& root, const // built BEFORE the log pipe opens, for the reason gitLogFileSets states: it runs a git probe of its own. const GitPathIndex byGitPath = gitPathIndexOfFiles( ing, onlyRoot ); - const std::string cmd = gitCmd( " -c core.quotepath=false -C " ) + shSingleQuote( root ) + " log --no-ext-diff --no-textconv " + kMergeDiffArgs + windowArgs - + "--name-only --format=tformat:__C__%x20%ct 2>/dev/null"; - std::FILE* pipe = os::popen( cmd.c_str(), "r" ); - if( !pipe ) + // Stream `git log --name-only` through the shared NUL-framed reader: one record per commit, its + // committer epoch in hand, its paths resolved against the index built above. + bool anyCommit = false; + forEachGitLogRecord( root, windowArgs, [ & ]( const GitLogRecord& rec ) { - return m; - } + anyCommit = true; + const std::int64_t epoch = rec.committerTs; + const std::int64_t ageSec = ( epoch > 0 && headEpoch > epoch ) ? ( headEpoch - epoch ) : 0; // clamped: never > 1 + const double curWeight = std::pow( 0.5, ( double( ageSec ) / 86400.0 ) / kChurnDecayHalfLifeDays ); - bool anyCommit = false; - double curWeight = 0.0; // this commit's decayed weight - std::int64_t curEpoch = 0; // this commit's committer epoch (F3: the newest one per file is kept) - std::vector cur; // this commit's resolved fileIds (dedup before tally) - const auto flush = [ & ]() - { + std::vector cur; // this commit's resolved fileIds (dedup before tally) + for( const std::string& gp : rec.paths ) + { + const std::uint32_t f = resolveGitPath( byGitPath, gp ); + if( f != UINT32_MAX ) + { + cur.push_back( f ); + } + } std::sort( cur.begin(), cur.end() ); cur.erase( std::unique( cur.begin(), cur.end() ), cur.end() ); if( cur.size() >= 1 && cur.size() <= maxFiles ) @@ -2091,44 +2201,14 @@ inline DecayedChurnMined gitLogDecayedFileMining( const std::string& root, const for( std::uint32_t f : cur ) { weights[f] += curWeight; - if( curEpoch > m.lastEpoch[f] ) { m.lastEpoch[f] = curEpoch; } + if( epoch > m.lastEpoch[f] ) { m.lastEpoch[f] = epoch; } // F3: the newest epoch per file is kept } } else if( cur.size() > maxFiles ) { ++m.mergeBombsSkipped; // the rule fired: disclosed on , never silently absorbed } - cur.clear(); - }; - std::string s; - while( readByteSafeLine( pipe, s ) ) // F6: THE line reader, not a char[4096] a long path can be split across - { - while( !s.empty() && ( s.back() == '\n' || s.back() == '\r' ) ) - { - s.pop_back(); - } - if( s.rfind( "__C__", 0 ) == 0 ) // new commit marker: "__C__ " - { - flush(); - anyCommit = true; - const std::int64_t epoch = ( s.size() > 6 ) ? std::strtoll( s.c_str() + 6, nullptr, 10 ) : 0; - const std::int64_t ageSec = ( epoch > 0 && headEpoch > epoch ) ? ( headEpoch - epoch ) : 0; // clamped: never > 1 - curWeight = std::pow( 0.5, ( double( ageSec ) / 86400.0 ) / kChurnDecayHalfLifeDays ); - curEpoch = epoch; - continue; - } - if( s.empty() ) - { - continue; - } - const std::uint32_t f = resolveGitPath( byGitPath, s ); - if( f != UINT32_MAX ) - { - cur.push_back( f ); - } - } - flush(); - os::pclose( pipe ); + } ); m.anyHistory = anyCommit; return m; } @@ -2430,11 +2510,11 @@ inline std::vector gitFileAuthors( }; const std::string rootSlash = root + "/"; - // git log -c --name-only --format=tformat:__C__%ae|%at 2>/dev/null (optionally scoped to one path with - // -- ): one "__C__email|unix-ts" header per commit, then its changed files, one per line, - // until a blank line / the next header. Mirrors gitCommitFileSets' exact invocation style. - std::string cmd = gitCmd( " -c core.quotepath=false -C " ) + shSingleQuote( root ) - + " log --no-ext-diff --no-textconv " + kMergeDiffArgs + "--name-only --format=tformat:__C__%ae\\|%at"; + // git log -c -z --name-only --format='%x00__C__%x00%ae%x00%at%x00%ct%x00' 2>/dev/null (optionally + // scoped to one path with `-- `): one record per commit through the shared NUL-framed + // reader, author email and author timestamp as their own fields. Mirrors every other consumer's + // single-pass invocation style. + std::string extraArgs; if( singleFile ) { const std::string& fp = ing.files[ onlyFileId ]; @@ -2443,65 +2523,47 @@ inline std::vector gitFileAuthors( { relPath = fp.substr( rootSlash.size() ); } - cmd += " -- " + shSingleQuote( relPath ); + // extraArgs rides at the END of the command (see gitLogRecordsCmd): a pathspec is only a pathspec + // when it comes last, and anything after `--` is read as a path or revision. The trailing space + // keeps the `2>/dev/null` redirect from gluing onto the quoted path. Both properties are + // load-bearing — without the first, git reads the following -z --name-only --format=… as path + // arguments and answers in its DEFAULT format, which the reader sees as a stream with no marker + // in it (measured as "git unavailable / no history" while the all-files form was correct). + extraArgs = " -- " + shSingleQuote( relPath ) + " "; } - cmd += " 2>/dev/null"; - std::FILE* pipe = os::popen( cmd.c_str(), "r" ); - if( !pipe ) + forEachGitLogRecord( root, extraArgs, [ & ]( const GitLogRecord& rec ) { - return {}; - } - - std::string curEmail; - std::int64_t curTs = 0; - bool haveCommit = false; - std::string s; - while( readByteSafeLine( pipe, s ) ) // F6: THE line reader, not a char[4096] a long path can be split across - { - while( !s.empty() && ( s.back() == '\n' || s.back() == '\r' ) ) + // The old line frame parsed `email|ts` out of one field and skipped the commit's files when + // either half was missing or the epoch did not parse. Under the NUL frame the fields arrive + // separately, so the same guard is an empty email or a non-positive epoch — git never stamps a + // real commit at or before the epoch, and a header that forged one is exactly what this frame + // exists to make impossible. + if( rec.authorEmail.empty() || rec.authorTs <= 0 ) + { + return; + } + std::string curEmail = rec.authorEmail; + for( char& c : curEmail ) { - s.pop_back(); + c = char( std::tolower( static_cast( c ) ) ); } - if( s.compare( 0, 5, "__C__" ) == 0 ) + const std::int64_t curTs = rec.authorTs; + for( const std::string& gp : rec.paths ) { - const std::string_view rest( s.data() + 5, s.size() - 5 ); - const std::size_t pipePos = rest.find( '|' ); - if( pipePos == std::string_view::npos || pipePos == 0 || pipePos + 1 >= rest.size() ) + const std::uint32_t f = resolve( gp ); + if( f == UINT32_MAX ) { - haveCommit = false; continue; } - curEmail = rest.substr( 0, pipePos ); - for( char& c : curEmail ) + perFile[ f ][ curEmail ].push_back( curTs ); + std::int64_t& newest = newestTsByFile[ f ]; + if( curTs > newest ) { - c = char( std::tolower( static_cast( c ) ) ); + newest = curTs; } - const std::string tsStr( rest.substr( pipePos + 1 ) ); - char* end = nullptr; - curTs = std::strtoll( tsStr.c_str(), &end, 10 ); - haveCommit = ( end != tsStr.c_str() ); - continue; } - if( s.empty() || !haveCommit ) - { - continue; // blank separator line, or malformed header — skip its files - } - - const std::uint32_t f = resolve( s ); - if( f == UINT32_MAX ) - { - continue; - } - - perFile[ f ][ curEmail ].push_back( curTs ); - std::int64_t& newest = newestTsByFile[ f ]; - if( curTs > newest ) - { - newest = curTs; - } - } - os::pclose( pipe ); + } ); std::vector result; result.reserve( perFile.size() ); diff --git a/test/churnjoincheck.sh b/test/churnjoincheck.sh index e495c2cba..4c5602dbb 100755 --- a/test/churnjoincheck.sh +++ b/test/churnjoincheck.sh @@ -849,14 +849,20 @@ fi # and those are safe BY SHAPE", and pinned the COUNT of readByteSafeLine adoptions at five. Both halves were the # trap. Counting adoptions cannot see a survivor (that is trap #10 stated exactly), and the count itself rots # the moment a reader is legitimately added — as it did here. The bar is now: no fixed-buffer reader survives -# ANYWHERE in the file (§9c asserts that from source, comments excluded), and at LEAST the five path readers -# go through the byte-safe one. A lower bound cannot rot upward; a survivor still fails §9c. +# ANYWHERE in the file (§9c asserts that from source, comments excluded), and the git-pipe readers go through a +# BYTE-SAFE reader — of EITHER family, because the git-porcelain-framing lane (test/gitframingcheck.sh) moved +# the five `--name-only` consumers onto the shared NUL-framed reader forEachGitLogRecord (which reads fields +# through readNulSafeField), and left the byte-safe LINE reader (readByteSafeLine, behind readWalkLine) +# carrying gitCommandLines' own output. A lower bound cannot rot upward; a survivor still fails §9c. LEFTOVER="$( grep -c 'char line\[ 4096 \]' "$ROOT/src/gitmine.h" || true )" SAFEREADERS="$( grep -c 'readByteSafeLine( pipe' "$ROOT/src/gitmine.h" || true )" +NULREADERS="$( grep -c 'forEachGitLogRecord( root' "$ROOT/src/gitmine.h" || true )" [ "${LEFTOVER:-1}" = 0 ] && ok "F6/completeness: no 'char line[ 4096 ]' path reader survives in gitmine.h" \ || no "F6/completeness: $LEFTOVER fixed-buffer path reader(s) still in gitmine.h — a long path can still be split there" -[ "${SAFEREADERS:-0}" -ge 5 ] && ok "F6/completeness: all five git-pipe PATH readers go through readByteSafeLine ($SAFEREADERS call sites: gitCommandLines, gitLogFileSets, gitLogNameOnlyRaw, gitFileCommitCountsInDayWindow, gitFileAuthors, + any later adopter)" \ - || no "F6/completeness: only $SAFEREADERS readByteSafeLine call site(s) in gitmine.h, expected at least the five path readers" +[ "${SAFEREADERS:-0}" -ge 1 ] && ok "F6/completeness: the line-shaped git-pipe reader keeps its byte-safe core (readByteSafeLine behind readWalkLine, $SAFEREADERS site — gitCommandLines' own output)" \ + || no "F6/completeness: no readByteSafeLine core site in gitmine.h — a git-pipe line reader lost its byte-safe read" +[ "${NULREADERS:-0}" -ge 5 ] && ok "F6/completeness: all five git-pipe PATH readers go through the shared byte-safe NUL reader forEachGitLogRecord ($NULREADERS call sites: gitLogFileSets, gitLogNameOnlyRaw, gitFileCommitCountsInDayWindow, gitLogDecayedFileMining, gitFileAuthors, + any later adopter)" \ + || no "F6/completeness: only $NULREADERS forEachGitLogRecord call site(s) in gitmine.h, expected at least the five path readers" # ── 9c. G3 — the migration claim is re-derived FROM SOURCE, not read from the comment that makes it ── # F6 recorded "every git-pipe reader in this file uses readByteSafeLine"; it was true at one of five. The fix From 97f49d6c9b73c99efee5c81a862d3e05f5c610c3 Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 15:57:53 +0100 Subject: [PATCH 03/15] =?UTF-8?q?gitmine:=20THE=20MERGE=20FIX=20=E2=80=94?= =?UTF-8?q?=20a=20name-bearing=20merge=20emits=20a=20prefix=20empty=20the?= =?UTF-8?q?=20first=20draft=20read=20as=20a=20boundary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Under `git log -c -z --name-only`, a merge commit's combined diff is preceded by ONE MORE empty field than a regular commit's name list: a regular commit follows its metadata directly with a '\n'-prefixed first path, while a merge follows with a prefix empty and then BARE paths. The first migration draft treated every empty field as the record boundary, so a merge's prefix empty closed the merge as a pathless record and its combined-diff files — precisely the files `-c` exists to recover, the "evil merge" content train 28 fixed at the COMMAND level — were dropped on the floor by the new parser. Measured on the pre-fix binary: mergechurncheck's merge-only file lost its --owners row entirely (mergeonly.cpp top=''), the merge's author vanished from shared.cpp's author set (2, want 3), and --for's churn= lens read '' for the merge's function. The fix is ONE field of lookahead: an empty field ends the record and opens the next iff the field AFTER it is the marker (or the stream ends); any other empty is a merge prefix, skipped but counted. The count also decides the first path's '\n': a regular commit's first path carries git's separator (empties == 0), a merge's arrives bare after its prefix (empties == 1), so the strip applies to exactly one of the two shapes. The one collision left is disclosed, not papered over: a merge whose FIRST combined path is a file literally named `__C__` reads as pathless, because its prefix empty is followed by the marker-spelled path — the same collision class the line frame had, at the same severity. Pinned in test/gitframingcheck.sh's commentary with its observable (the merge's files lose their attribution; a pathless phantom record with no metadata is emitted and every consumer no-ops on it). The full fix is a format sentinel change — follow-on spec material. --- src/gitmine.h | 92 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 66 insertions(+), 26 deletions(-) diff --git a/src/gitmine.h b/src/gitmine.h index d6f076c5d..ca04656db 100644 --- a/src/gitmine.h +++ b/src/gitmine.h @@ -746,15 +746,24 @@ inline GitCommandLines gitCommandLines( const std::string& cmd ) // RAW (no C-quoting), and the marker is a WHOLE-FIELD sentinel a path can never spell. // // The pinned byte layout (specs/git-porcelain-framing/evidence/layout-pin.md; the prototype command -// that produced it is in the spec's Implementation Notes): +// that produced it is in the spec's Implementation Notes; re-pinned by test/gitframingcheck.sh): // // \0 __C__ \0 \0 \0 [ \0 ] [ \n | ] \0 \0 … // // Every record opens with an empty field, then the marker, then its three metadata fields. A commit // with paths follows the metadata with a '\n'-prefixed first path (git's separator); a path-less -// commit stops after its metadata. The next record's opening empty field is the record boundary — -// and an empty field can only come from the format, because paths are never empty. A field that ends +// commit stops after its metadata. A MERGE follows with one MORE empty field — the prefix empty — +// and then BARE paths with no '\n', which is what a combined diff emits; a clean merge stops after +// its prefix empty. The next record's opening empty field is the record boundary. A field that ends // at end-of-stream without its delimiter still counts; readNulSafeField returns it. +// +// The boundary rule, with ONE field of lookahead: an empty field ends the current record and opens +// the next iff the field after it is the marker (or the stream ends). Any other empty field is a +// merge prefix and is skipped — that is what keeps a merge's combined-diff files, the ones the first +// migration draft dropped, attached to the merge that introduced them. The one collision left is the +// same class the line frame had: a merge whose FIRST combined path is a file literally named `__C__` +// reads as a pathless record, because its prefix empty is followed by the marker-spelled path. That +// boundary is disclosed here and pinned in the gate rather than papered over. struct GitLogRecord { std::string authorEmail; @@ -774,14 +783,19 @@ struct GitLogWalk int exitCode = 0; }; -// The NUL-framed field iterator over one `git log -z` stream: every --format field and every -// --name-only path arrives as one readNulSafeField call, and a zero-length field — the record -// boundary — is a successful read, not the end of the stream. +// One field of lookahead over a NUL-framed stream: `next` is readNulSafeField or the held field, which +// the boundary rule in forEachGitLogRecord parks here while it decides what an empty field means. struct NulFieldStream { - std::FILE* pipe = nullptr; + std::FILE* pipe = nullptr; + std::string held; + bool hasHeld = false; - bool next( std::string& field ) { return readNulSafeField( pipe, field ); } + bool next( std::string& field ) + { + if( hasHeld ) { field = std::move( held ); hasHeld = false; return true; } + return readNulSafeField( pipe, field ); + } }; // The marker as a whole field. A path may carry these bytes; only a field that IS this string opens a record. @@ -838,35 +852,57 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin return walk; } - NulFieldStream fields{ pipe }; + NulFieldStream fields{ pipe, {}, false }; std::string field; + std::string lookahead; + bool haveLookahead = false; + bool atEnd = false; GitLogRecord cur; - bool inRecord = false; // a record is open (its opening empty field was seen) - bool sawMarker = false; // ...and its marker was read: a record without one is never visited - bool wantMarker = false; // the next field must be the marker - int metaLeft = 0; // metadata fields still to read for this record - bool havePath = false; // this record has taken its first path + bool inRecord = false; // a record is open (its opening empty field was seen) + bool sawMarker = false; // ...and its marker was read: a record without one is never visited + bool wantMarker = false; // the next field must be the marker + int metaLeft = 0; // metadata fields still to read for this record + int empties = 0; // empty fields since this record's metadata (0 = none, 1 = a merge prefix) + bool havePath = false; // this record has taken its first path const auto closeRecord = [ & ]() { if( inRecord && sawMarker ) { visit( cur ); } - cur = GitLogRecord{}; - sawMarker = false; + cur = GitLogRecord{}; + sawMarker = false; }; - while( fields.next( field ) ) + while( !atEnd ) { if( gitWalksAbandoned() ) { break; } // the walk readers' abandon contract: stop at the next field; pclose ends the child + if( haveLookahead ) { field = std::move( lookahead ); haveLookahead = false; } + else if( !fields.next( field ) ) { atEnd = true; break; } + if( field.empty() ) { - closeRecord(); - inRecord = true; - wantMarker = true; - metaLeft = 3; - havePath = false; + // One field of lookahead decides what this empty IS: the record boundary (the next field is + // the marker, or the stream ends here) or a merge's prefix empty (anything else follows). + std::string next; + bool haveNext = haveLookahead ? ( next = std::move( lookahead ), haveLookahead = false, true ) : fields.next( next ); + if( !haveNext || next == kGitLogMarker ) + { + closeRecord(); + inRecord = true; + wantMarker = true; + metaLeft = 3; + empties = 0; + havePath = false; + if( haveNext ) { lookahead = std::move( next ); haveLookahead = true; } + } + else + { + ++empties; // a merge prefix (or the record's own close): skipped, but counted + lookahead = std::move( next ); haveLookahead = true; + } continue; } + if( wantMarker ) { wantMarker = false; @@ -881,10 +917,13 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin --metaLeft; continue; } + + // A path. The first path of a REGULAR commit arrives with git's '\n' separator (no empty was + // counted since the metadata); a merge's first path arrives bare after its prefix empty. if( !havePath ) { havePath = true; - if( field.front() == '\n' ) { field.erase( 0, 1 ); } // git's separator before a commit's first path + if( empties == 0 && !field.empty() && field.front() == '\n' ) { field.erase( 0, 1 ); } } cur.paths.push_back( std::move( field ) ); } @@ -2510,10 +2549,11 @@ inline std::vector gitFileAuthors( }; const std::string rootSlash = root + "/"; - // git log -c -z --name-only --format='%x00__C__%x00%ae%x00%at%x00%ct%x00' 2>/dev/null (optionally + // git log -c -z --name-only --format='%x00__C__%x00%ae%x00%at%x00%ct' 2>/dev/null (optionally // scoped to one path with `-- `): one record per commit through the shared NUL-framed - // reader, author email and author timestamp as their own fields. Mirrors every other consumer's - // single-pass invocation style. + // reader, author email and author timestamp as their own fields. NO trailing %x00 — the last + // field runs to the stream's end, and the boundary rule reads the empty that FOLLOWS it. + // Mirrors every other consumer's single-pass invocation style. std::string extraArgs; if( singleFile ) { From 4c0aa6736bf6e3e589a9b420a43b78fe189414d3 Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 18:20:32 +0100 Subject: [PATCH 04/15] =?UTF-8?q?test:=20the=20git-log=20FRAMING=20gate=20?= =?UTF-8?q?=E2=80=94=20a=20committed=20filename=20can=20never=20be=20read?= =?UTF-8?q?=20as=20commit=20metadata?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate the porcelain-framing lane promised (commit 2 said "test/gitframingcheck.sh re-pins them with a raw-stream digest"; this is that file). Its fixture is an 11-commit history whose files are named to attack the old LINE frame: `__C__`, `__C__x|1694000000.cpp`, `__C__ 1694000000.cpp`, `__C__.cpp`, `q"uote.cpp`, `sp ace.cpp`, `back\slash.cpp`, `pi|pe.cpp`, `--dashdash.cpp`, `ab.cpp` — with the hostile names sorting BEFORE the indexed files they must not corrupt. It carries both merge shapes the boundary rule turns on: a name-bearing merge whose first combined path is an ORDINARY file (the one-field lookahead must attach it) and a merge whose first combined path is the marker-spelled file itself — the disclosed collision, pinned rather than papered over. Red first: on the pre-change binary the gate fails 12 arms, with the forgery measured where it happened (files="8" of 11 indexed, five files owned by an author email that appears in no commit, zeta.cpp's top owner flipped to quiet2@x.com, the newline file's churn lost entirely). Arms: (a) the spoof is neutralized — every row names its true author and the forged email appears nowhere; (b) the newline filename round-trips as ONE path (churn=1, owned, no phantom rows for the split halves); (c) the exact-marker file is a path, never a record opener; (d) the honest tree is byte-identical to a golden recorded from the PRE-CHANGE binary over owners/hotspots/cochange; (e) the marker literal lives only inside the shared helper. Then the raw-stream digest plus the 13 NUL-delimited marker-spelled fields (11 record openers + 2 paths), the merge shapes (with the two miners' disagreement on the collision pinned as BY DESIGN), author-not-committer, non-vacuity, determinism, an oracle that re-derives the tool's churn from git's own stream, the 2.43.0 git floor, and a shimmed git that rejects -z: the reader must degrade HONESTLY — non-zero exit, no fabricated rows, "git unavailable / no history" on stderr — rather than answer from an unframable stream. The fixtures are fresh mktemp paths every run, and that is load-bearing beyond tidiness: the tool memoizes its history walks under $TMPDIR/ripwire keyed on (repo, HEAD, window) with a path-derived repo key, so a fixed fixture path would let a previous run's cached churn answer this run's questions. A fresh path is a cold cache: every number in the gate is computed, never remembered. Registered in regression.sh's absorb loop (between gitenvhermeticcheck and githardencheck). The published gate count is the generator's build product, so docs/gatecount_build.py rewrote the three marked sites 703 -> 704; gatecountcheck and manifestcheck agree. --- README.md | 4 +- docs/EVALS.md | 6 +- present/deck5_ripwire_build.js | 6 +- test/gitframingcheck.sh | 490 ++++++++++++++++++++++++++++++++ test/gitframingcheck_golden.txt | 6 + test/regression.sh | 2 +- 6 files changed, 505 insertions(+), 9 deletions(-) create mode 100755 test/gitframingcheck.sh create mode 100644 test/gitframingcheck_golden.txt diff --git a/README.md b/README.md index df8542c7c..f07180754 100644 --- a/README.md +++ b/README.md @@ -2120,9 +2120,9 @@ wrong, and it has. These are the results that say so, all in-tree, all published ### In the tests
-703 gate scripts, five contracts no unit test can hold, and the house rule: write the gate before the code it measures +704 gate scripts, five contracts no unit test can hold, and the house rule: write the gate before the code it measures -`test/regression.sh` names **703 gate scripts** and is the authoritative list; +`test/regression.sh` names **704 gate scripts** and is the authoritative list; `python3 test/pargates.py . ./build/ripwire -j 6` runs the same set in parallel. On top of them sit the contracts that do not fit a unit test: two runs byte-identical, warm output identical to cold, output that pipes clean through `xmllint --noout`, a sanitizer build with `-fno-sanitize-recover=all`, and a diff --git a/docs/EVALS.md b/docs/EVALS.md index a77007342..ab0849671 100644 --- a/docs/EVALS.md +++ b/docs/EVALS.md @@ -21,7 +21,7 @@ section, and it is not an afterthought. | **Co-change / known-item evals** | `--eval`, `--eval-retrieval` (see `bench/ANSWERQUALITY.md`) | Whether the tool surfaces the other files a real historical commit touched; and known-item retrieval across four rankers. | | **Ensemble calibration harness** | `bench/ensemblecal/` | Whether `--ensemble`'s four evidence families are actually orthogonal, how often each fires, how stable each is across commits — and the preset ladder derived from that (§9). | | **Differential argv harness** | `test/argvdiffcheck.sh` | That a refactor changed *nothing observable*: two binaries, every argv vector, stdout + stderr + exit code byte-identical. | -| **The gate suite** | `test/regression.sh`, `test/pargates.py` | 703 gate scripts plus the determinism, cache-transparency and golden contracts. | +| **The gate suite** | `test/regression.sh`, `test/pargates.py` | 704 gate scripts plus the determinism, cache-transparency and golden contracts. | | **`--quality-delta`** | `src/quality.h` | Ten measured code-quality failure modes, reported only where a change made them worse. | ### The labeling protocol (why the held-out eval is allowed to disagree with the ranker) @@ -5888,7 +5888,7 @@ copy here would be exactly the dialect divergence that gate exists to catch. Com tags, wrap, stable-order defaults), seven individually invoked standalone gates (`g1freshcheck`, `skillscan`, `htmlexport`, `compresscheck`, `handoffcheck`, `releaseinstallcheck`, `taskroutecheck`), and a single loop -naming **703 gate scripts**, all of which exist on disk. +naming **704 gate scripts**, all of which exist on disk. `python3 test/pargates.py . ./build/ripwire -j 6` runs the same scripts in parallel so a full verification fits in one sitting. It does not modify `regression.sh`. @@ -7005,7 +7005,7 @@ Listed because the reason is more useful than the silence. shipped**. See `bench/locbench/anchorhop_calib.json`. The mention anchor's reproducible numbers are the ablations in §4. - **A single round gate-count.** Two in-tree numbers disagree (`test/pargates.py`'s docstring says - ~210; `test/argvdiffcheck.sh` says 200+), while the loop in `test/regression.sh` names 703. The + ~210; `test/argvdiffcheck.sh` says 200+), while the loop in `test/regression.sh` names 704. The loop is the authority; the stale docstrings are a known drift. Since 2026-09-10 the number is not written by hand anywhere: `docs/gatecount_build.py` derives it from the loop and rewrites every published site, `test/gatecountcheck.sh` fails if any of them drifts, and `test/manifestcheck.sh` diff --git a/present/deck5_ripwire_build.js b/present/deck5_ripwire_build.js index ac507279a..04c345674 100644 --- a/present/deck5_ripwire_build.js +++ b/present/deck5_ripwire_build.js @@ -1214,7 +1214,7 @@ function storyCards(s, { kick, head, stories, footText }){ kicker(s, "// how it stays true", AMBER); title(s, "Proven, not promised"); const cards = [ - ["703 gate scripts", "the suite runs on every push — plus determinism, cache-transparency and golden contracts; the gate count itself is gated against the runner's own loop"], // gatecount + ["704 gate scripts", "the suite runs on every push — plus determinism, cache-transparency and golden contracts; the gate count itself is gated against the runner's own loop"], // gatecount ["byte-identical, always", "two runs over the same tree produce the same bytes; warm equals cold. Enforced in CI, twice — Release AND a plain flavour, because NDEBUG once blinded a whole class of checks"], ["differential refactoring", "a refactor must prove it changed nothing observable: two binaries, hundreds of argv vectors, stdout + stderr + exit codes byte-identical"], ["held-out labels, authored blind", "eval labels were written by reading source before the ranker ever ran on them — so the eval is allowed to say the ranker is wrong. It has."], @@ -1238,7 +1238,7 @@ function storyCards(s, { kick, head, stories, footText }){ title(s, "Claims you can trust, because we publish what failed", { size: 32 }); card(s, MX, 1.72, 3.86, 1.72); - stat(s, "703", "gate scripts named by test/regression.sh — and the COUNT itself is gated against the runner's own loop, so it cannot go stale quietly", // gatecount + stat(s, "704", "gate scripts named by test/regression.sh — and the COUNT itself is gated against the runner's own loop, so it cannot go stale quietly", // gatecount MX+0.15, 1.86, 3.56, CYAN, { bsize: 42, bh: 0.66, lsize: 9.5 }); card(s, 4.68, 1.72, 3.86, 1.72, CARD2); stat(s, "8", "registered NEGATIVES — changes built, gated green, measured against a band written before the code, and reverted rather than tuned", @@ -1540,7 +1540,7 @@ function storyCards(s, { kick, head, stories, footText }){ ["192 long flags · 36 slides", "bash test/deckclaimcheck.sh"], ["every --flag named here exists", "bash test/deckcheck.sh"], ["74.7% fewer element bytes", "bash test/showcasecapturecheck.sh"], - ["703 gate scripts", "bash test/manifestcheck.sh"], // gatecount + ["704 gate scripts", "bash test/manifestcheck.sh"], // gatecount ["49 repos · 71 papers · 237 surveyed","bash test/readmedriftcheck.sh"], ["the ten moments, any row", "ripwire . --callers=SYM | wc -c"], ["the head-to-head table", "bench/headtohead/r4-2026-08-06/"], diff --git a/test/gitframingcheck.sh b/test/gitframingcheck.sh new file mode 100755 index 000000000..8be90fa1d --- /dev/null +++ b/test/gitframingcheck.sh @@ -0,0 +1,490 @@ +#!/usr/bin/env bash +# gitframingcheck.sh — the git-log FRAMING gate: a committed filename can never be read as commit +# metadata, and the NUL frame that guarantees it stays pinned. +# +# THE DEFECT (audit finding #4, Medium). Every `git log --name-only` consumer in src/gitmine.h framed +# its stream with a LINE-based reader and a `__C__` marker matched as a line PREFIX — three spellings +# across five sites — so a path whose name begins with the marker bytes was read as commit metadata. +# Measured on the hostile fixture below with the PRE-CHANGE binary: +# files="8" of 11 indexed — the four marker-prefixed names (`__C__`, `__C__ 1694000000.cpp`, +# `__C__.cpp`, `__C__x|1694000000.cpp`) read as forged headers instead of paths +# five files owned by top="x", an author email that appears in NO commit +# (`__C__x|1694000000.cpp` sorts before lowercase a..z, so every path after it inherits the +# forgery; `--dashdash.cpp` and `Alpha.cpp`, sorting before it, keep their true author) +# zeta.cpp carrying x as a co-author (authors=3), its top owner flipped to quiet2@x.com — the +# forgery changes the ANSWER, not a detail +# +# THE FIX (lane/git-porcelain-framing-v2). One reader parses the stream for all five consumers: +# `git log -z --name-only --format=%x00__C__%x00%ae%x00%at%x00%ct`. Fields and paths are +# NUL-terminated and RAW (core.quotepath=false), and the marker is a WHOLE-FIELD sentinel a path can +# never spell. An empty field — which only the format can emit, because paths are never empty — is +# the record boundary, read with ONE field of lookahead so a merge's prefix empty is not mistaken +# for one. The five consumers (gitLogFileSets, gitLogNameOnlyRaw, gitFileCommitCountsInDayWindow, +# gitLogDecayedFileMining, gitFileAuthors) became visitors over forEachGitLogRecord. +# +# THE PINNED LAYOUT (specs/git-porcelain-framing/evidence/layout-pin.md, derived empirically — the +# '\n' before a commit's first path, the merge prefix empty and the end-of-stream unterminated field +# are NOT documented git behaviour): +# \0 __C__ \0 \0 \0 [ \0 ] [ \n | ] \0 \0 ... +# regular commit: metadata, then a '\n'-prefixed first path. empty commit: metadata only. +# merge: metadata, a prefix empty, then BARE paths. clean merge: metadata + prefix empty. +# MINIMUM GIT 2.43.0 (arm 8 refuses below it): a git that changes any of the above fails arm 2 +# loudly instead of silently corrupting the first path. +# +# ARMS +# (a) the spoof is neutralized — every --owners row on the hostile fixture names its TRUE author, +# and no forged-author row exists anywhere. +# (b) the newline filename round-trips — ab.cpp is exactly one path, churn=1, owned. +# (c) the exact-marker file is a path — a file named exactly `__C__` never opens a record; the +# indexed file after it in byte order keeps its attribution. +# (c2) the single-file pathspec seam — --owners=SYM appends a pathspec to the shared command, and a +# pathspec glued to -z answers EMPTY, which reads as "git unavailable / no history" rather than +# as a broken command. The pin for the one space that separates them. +# (d) the honest tree is byte-identical to the golden RECORDED FROM THE PRE-CHANGE BINARY — the +# "no behavior change on honest trees" claim, over --owners/--hotspots/--cochange. +# (e) the marker literal lives ONLY inside the shared helper — five consumer bodies, zero hits. +# 2. the raw-stream digest + structural count (11 records, 13 marker-spelled byte sequences). +# 3. the merge shapes: an ordinary name-bearing merge's files are ATTACHED (the lookahead is +# load-bearing — a parser without it drops them); the DISCLOSED collision is pinned. +# 4. author-not-committer: ownership answers with the AUTHOR, never the committer. +# 5. non-vacuity: the fixture is what the arms claim (12 tracked, 11 indexed, the extension-less +# `__C__` unindexed BY DESIGN per layout-pin Finding 1). +# 6. determinism: two runs byte-identical. +# 7. the oracle: the tool's numbers re-derive from its own command (trap #12). +# 8. the git floor, and a shimmed git that rejects -z degrades honestly — no fabricated rows. +# +# THE DISCLOSED COLLISION (not papered over; full fix = a format sentinel change, follow-on spec +# material): a merge whose FIRST combined path is a file literally named `__C__` reads as pathless — +# its prefix empty is followed by the marker-spelled path, which the boundary rule cannot tell from +# a real record boundary. The fixture carries exactly such a merge (MERGE, resolving `__C__` and +# `__C__.cpp`), and arm 3 pins its observable: the merge's files lose their attribution, and a +# pathless phantom record (whose "author" is the marker-spelled path) is emitted that every consumer +# no-ops on. Note the two miners disagree there BY DESIGN of the disclosure: --for's churn (the +# migrated reader) reads 3 for `__C__.cpp`, while --hotspots' churn rides the unmigrated +# gitCommandLines miner and reads 4. +# +# EXPIRY: like test/mergechurncheck.sh this fixture hardcodes 2026-06 dates; the churn verbs mine a +# HEAD-anchored 12-month window, so the numbers are stable, but a future reader moving the dates +# must re-derive every literal here. The digest pins the exact stream. +# +# Usage: test/gitframingcheck.sh # build/ripwire +# test/gitframingcheck.sh asan/ripwire # positional seam +# RIPWIRE_BIN=asan/ripwire test/gitframingcheck.sh # env seam +set -u +ROOT="$( cd "$( dirname "$0" )/.." && pwd )" +. "$ROOT/test/lib/clean-env.sh" +BIN="${1:-${RIPWIRE_BIN:-$ROOT/build/ripwire}}" # BOTH seams — positional and env +[ "${BIN#/}" = "$BIN" ] && BIN="$ROOT/$BIN" +GOLD="$ROOT/test/gitframingcheck_golden.txt" +SRC="$ROOT/src/gitmine.h" +fail=0 +ok(){ printf ' PASS %s\n' "$*" || { fail=1; printf ' FAIL could not write the PASS line for: %s\n' "$*"; }; return 0; } +no(){ printf ' FAIL %s\n' "$*"; fail=1; } + +[ -x "$BIN" ] || { echo "no ripwire binary at $BIN — build first (cmake --build build -j)"; exit 2; } +command -v git >/dev/null 2>&1 || { echo "git required"; exit 2; } +REAL_GIT="$( command -v git )" +echo "gitframingcheck: BIN=$BIN ROOT=$ROOT git=$( git --version )" + +TMP="$( mktemp -d )"; H="$( mktemp -d )"; G="$( mktemp -d )" +trap 'rm -rf "$TMP" "$H" "$G"' EXIT +# The fixtures are FRESH mktemp paths every run, and that is load-bearing beyond tidiness: the tool +# memoizes its history walks under $TMPDIR/ripwire keyed on (repo, HEAD, window) — the repo key is +# path-derived — so a fixed fixture path would let a previous run's cached churn answer this run's +# questions. A fresh path is a cold cache: every number below is computed, never remembered. + +mkfn(){ printf 'int %s( int x )\n{\n if( x > 1 ) return x + 1;\n return x - 1;\n}\n' "$1"; } +D(){ export GIT_AUTHOR_DATE="$1" GIT_COMMITTER_DATE="$1"; } +ID(){ export GIT_AUTHOR_NAME="$1" GIT_AUTHOR_EMAIL="$2" GIT_COMMITTER_NAME="$3" GIT_COMMITTER_EMAIL="$4"; } +UC(){ unset GIT_AUTHOR_DATE GIT_COMMITTER_DATE GIT_AUTHOR_NAME GIT_AUTHOR_EMAIL GIT_COMMITTER_NAME GIT_COMMITTER_EMAIL; } +# attr_of FILE PATHSUFFIX ATTR — the ATTR value of the row whose p= ends with PATHSUFFIX ("" if no row). +# p= is XML-escaped in the output, so ab.cpp spells p="a b.cpp" and q"uote.cpp spells +# p="q"uote.cpp"; callers pass the ESCAPED spelling, which is also the stable one. +psel(){ awk -v s="$1" '{ if( match( $0, /p="[^"]*"/ ) ) { v = substr( $0, RSTART + 3, RLENGTH - 4 ); + if( v == s || ( length(v) > length(s) && substr( v, length(v) - length(s) ) == "/" s ) ) print } }'; } +attr_of(){ tr '>' '\n' < "$1" | psel "$2" | grep -oE "$3=\"[^\"]*\"" | head -1 | sed 's/^[^"]*"//; s/"$//'; } +churn_of_fn(){ tr '>' '\n' < "$1" | grep -F "n=\"$2\"" | grep -oE 'churn="[0-9]+"' | head -1 | tr -cd '0-9'; } +# code_only FILE — FILE with comment-only lines dropped, for "the literal appears in CODE" counts. +# This codebase's block-comment interior lines start with '*', and both code sites carrying the marker +# are plain code lines, so the count is exact for the pins below. +code_only(){ awk '{ l=$0; sub(/^[ \t]+/,"",l); if( l ~ /^\/\// || l ~ /^\*/ || l ~ /^\/\*/ ) next; print }' "$1"; } + +# ── THE HOSTILE FIXTURE ───────────────────────────────────────────────────────────────────────── +# History, read commit by commit (these readings ARE the literals asserted below — never computed +# from the tool, never from the same git flag the code uses): +# c1 Real 2026-06-01 creates the 12-file battery: Alpha.cpp, zeta.cpp, and the +# hostile names `__C__`, `__C__ 1694000000.cpp`, `__C__.cpp`, +# `__C__x|1694000000.cpp`, `q"uote.cpp`, `sp ace.cpp`, `back\slash.cpp`, +# `pi|pe.cpp`, `--dashdash.cpp`, `ab.cpp +# c2 author Real, COMMITTER Committer 2026-06-03 — appends zeta.cpp +# (the author-not-committer arm: author date 06-02, committer date 06-03) +# s1 Sidedev 2026-06-04 on branch `side`: modifies `__C__` and `__C__.cpp` +# m1 Real 2026-06-05 on main: modifies the same two differently (a conflict on both) +# MERGE author Real, COMMITTER Mergecommitter 2026-06-07 — resolves BOTH +# with content neither parent had, so the combined diff is +# [`__C__`, `__C__.cpp`] and its FIRST path is the marker-spelled file: +# THE DISCLOSED COLLISION. The merge reads as pathless. +# c3 Real 2026-06-08 --allow-empty: the path-less record +# q1 Quietdev 2026-06-09 on branch `quiet`: modifies Alpha.cpp +# m2 Real 2026-06-10 on main: modifies Alpha.cpp the other way +# q2 Quietdev 2026-06-11 on branch `quiet2`: appends zeta.cpp +# CLEAN Real 2026-06-12 `git merge -s ours --no-ff quiet2`: a real two-parent merge whose +# combined diff is EMPTY (the clean-merge shape) +# M2 author Real, COMMITTER Mergecommitter 2026-06-13 — merges `quiet`, resolves Alpha.cpp +# with new content: a name-bearing merge whose first combined path is an +# ORDINARY file. The lookahead attaches it; a parser without one drops it. +mkdir -p "$H" +git -C "$H" init -q -b main +C(){ git -C "$H" add -A >/dev/null; git -C "$H" commit -qm "$1"; } + +# layout-pin Finding 1: an unindexed hostile name is invisible to every miner, so the forgery is only +# observable where a hostile name sorts BEFORE an indexed file. `__C__` (0x5F) sorts before lowercase +# a..z; `Alpha.cpp` (0x41) sorts before it. The extension-less `__C__` is unindexed BY DESIGN — its +# path-level survival is pinned by the digest (arm 2), and its danger through the files after it. +ID Real real@x.com Real real@x.com; D 2026-06-01T12:00:00 +mkfn alphaFn > "$H/Alpha.cpp" +mkfn zetaFn > "$H/zeta.cpp" +mkfn cFn > "$H/__C__" +mkfn spFn > "$H/__C__ 1694000000.cpp" +mkfn dotFn > "$H/__C__.cpp" +mkfn spoofFn > "$H/__C__x|1694000000.cpp" +mkfn qFn > "$H/q\"uote.cpp" +mkfn spFn2 > "$H/sp ace.cpp" +mkfn bsFn > "$H/back\\slash.cpp" +mkfn piFn > "$H/pi|pe.cpp" +mkfn ddFn > "$H/--dashdash.cpp" +mkfn nlFn > "$H/a"$'\n'"b.cpp" +C c1 + +ID Real real@x.com Committer committer@x.com +D 2026-06-02T12:00:00; export GIT_COMMITTER_DATE="2026-06-03T09:00:00" # author 06-02, committer 06-03 +printf '// c2 edit\n' >> "$H/zeta.cpp" +C c2 +unset GIT_COMMITTER_DATE + +git -C "$H" checkout -q -b side +ID Sidedev side@x.com Sidedev side@x.com; D 2026-06-04T12:00:00 +printf 'int cFn( int x )\n{\n return x + 2;\n}\n' > "$H/__C__" +printf 'int dotFn( int x )\n{\n return x + 3;\n}\n' > "$H/__C__.cpp" +C s1 + +git -C "$H" checkout -q main +ID Real real@x.com Real real@x.com; D 2026-06-05T12:00:00 +printf 'int cFn( int x )\n{\n return x + 4;\n}\n' > "$H/__C__" +printf 'int dotFn( int x )\n{\n return x + 5;\n}\n' > "$H/__C__.cpp" +C m1 + +ID Real real@x.com Mergecommitter mergecommitter@x.com; D 2026-06-06T12:00:00 +export GIT_COMMITTER_DATE="2026-06-07T08:00:00" +git -C "$H" merge --no-commit --no-ff side >/dev/null 2>&1 || true # conflicts on both by construction +printf 'int cFn( int x )\n{\n return x + 6;\n}\n' > "$H/__C__" +printf 'int dotFn( int x )\n{\n if( x > 1 ) return x + 7;\n return x - 7;\n}\n' > "$H/__C__.cpp" +C "merge side (collision: first combined path is the file named __C__)" +unset GIT_COMMITTER_DATE + +ID Real real@x.com Real real@x.com; D 2026-06-08T12:00:00 +git -C "$H" commit -q --allow-empty -m c3 + +git -C "$H" checkout -q -b quiet +ID Quietdev quiet@x.com Quietdev quiet@x.com; D 2026-06-09T12:00:00 +printf 'int alphaFn( int x )\n{\n return x + 8;\n}\n' > "$H/Alpha.cpp" +C q1 + +git -C "$H" checkout -q main +ID Real real@x.com Real real@x.com; D 2026-06-10T12:00:00 +printf 'int alphaFn( int x )\n{\n return x + 9;\n}\n' > "$H/Alpha.cpp" +C m2 + +git -C "$H" checkout -q -b quiet2 +ID Quietdev quiet2@x.com Quietdev quiet2@x.com; D 2026-06-11T12:00:00 +printf '// q2 edit\n' >> "$H/zeta.cpp" +C q2 +git -C "$H" checkout -q main +ID Real real@x.com Real real@x.com; D 2026-06-12T12:00:00 +git -C "$H" merge -s ours --no-commit --no-ff quiet2 >/dev/null 2>&1 || true # clean: main's tree wins +C "merge quiet2 (clean: -s ours, empty combined diff)" + +ID Real real@x.com Mergecommitter mergecommitter@x.com; D 2026-06-13T12:00:00 +git -C "$H" merge --no-commit --no-ff quiet >/dev/null 2>&1 || true # conflicts on Alpha.cpp by construction +printf 'int alphaFn( int x )\n{\n if( x > 1 ) return x + 10;\n return x - 10;\n}\n' > "$H/Alpha.cpp" +C "merge quiet (name-bearing: resolves Alpha.cpp, first combined path is ordinary)" +UC + +# ── THE HONEST FIXTURE (arm d) ───────────────────────────────────────────────────────────────────── +# A tree with NO hostile names. The claim under test: the new reader changes NOTHING here — the golden +# was recorded from the PRE-CHANGE binary over exactly this history. The fixture is byte-deterministic +# (every date, identity and message pinned), so its HEAD sha — and the golden's at= — are stable; a +# reader who moves a date must re-record the golden, which is the EXPIRY note above made concrete. +# b1 Basedev 2026-06-01 Alpha.cpp, zeta.cpp, shared.cpp +# s1 Sidedev 2026-06-02 on `side`: adds sidefile.cpp, appends shared.cpp +# m1 Basedev 2026-06-03 on main: appends shared.cpp differently (a conflict on both sides) +# MERGE author Mergeauthor, COMMITTER Mergecommitter 2026-06-04/05 — a name-bearing merge: resolves +# the conflict with new content and adds mergeonly.cpp; its first combined path is ORDINARY +# c2 Basedev 2026-06-06 appends Alpha.cpp and zeta.cpp +mkdir -p "$G" +git -C "$G" init -q -b main +CG(){ git -C "$G" add -A >/dev/null; git -C "$G" commit -qm "$1"; } +ID Basedev base@x.com Basedev base@x.com; D 2026-06-01T12:00:00 +mkfn baseFn > "$G/Alpha.cpp" +mkfn zetaFn > "$G/zeta.cpp" +mkfn sharedFn > "$G/shared.cpp" +CG b1 +git -C "$G" checkout -q -b side +ID Sidedev side@x.com Sidedev side@x.com; D 2026-06-02T12:00:00 +mkfn branchFn > "$G/sidefile.cpp" +printf '// s1\n' >> "$G/shared.cpp" +CG s1 +git -C "$G" checkout -q main +ID Basedev base@x.com Basedev base@x.com; D 2026-06-03T12:00:00 +printf '// m1\n' >> "$G/shared.cpp" +CG m1 +ID Mergeauthor mergeauthor@x.com Mergecommitter mergecommitter@x.com; D 2026-06-04T12:00:00 +export GIT_COMMITTER_DATE="2026-06-05T08:00:00" # author 06-04, committer 06-05 +git -C "$G" merge --no-commit --no-ff side >/dev/null 2>&1 || true +mkfn mergeOnlyFn > "$G/mergeonly.cpp" +mkfn sharedFn > "$G/shared.cpp"; printf '// resolved in the merge itself\n' >> "$G/shared.cpp" +CG "merge side (name-bearing)" +unset GIT_COMMITTER_DATE +ID Basedev base@x.com Basedev base@x.com; D 2026-06-06T12:00:00 +printf '// c2\n' >> "$G/Alpha.cpp"; printf '// c2\n' >> "$G/zeta.cpp" +CG c2 +UC + +# ── THE TOOL'S ANSWERS ON THE HOSTILE FIXTURE ────────────────────────────────────────────────────── +"$BIN" --owners --detail=1 --limit=60 "$H" > "$TMP/owners.xml" 2>"$TMP/owners.err"; owners_rc=$? +"$BIN" --hotspots --limit=60 "$H" > "$TMP/hotspots.xml" 2>"$TMP/hotspots.err" +for fn in alphaFn dotFn zetaFn nlFn spoofFn; do + "$BIN" --for=$fn "$H" > "$TMP/for-$fn.xml" 2>/dev/null +done + +# ── ARM (a): the spoof is neutralized ────────────────────────────────────────────────────────────── +echo "arm a: the spoof is neutralized on the hostile fixture" +if [ "$owners_rc" = 0 ]; then ok "--owners exits 0 on the hostile fixture" +else no "--owners exits $owners_rc: $( head -1 "$TMP/owners.err" )"; fi +# rows start with `' '\n' < "$TMP/owners.xml" | grep 'b.cpp has churn=1 — one path, one commit (a line frame splits it into 'a' and 'b.cpp')" \ + || no "ab.cpp churn='$nl_churn', want 1" +nl_top=$( attr_of "$TMP/owners.xml" 'a b.cpp' top ) +[ "$nl_top" = "real@x.com" ] && ok "ab.cpp is owned by its true author" \ + || no "ab.cpp is owned by '$nl_top'" +halves=$( tr '>' '\n' < "$TMP/owners.xml" | grep -cE '-z --name-only ...`, so ANY extraArgs a consumer appends +# must carry its own trailing space. gitFileAuthors' --owners=SYM form appends a pathspec (`-- `); +# without that space the shell reads the pathspec and the -z as ONE word (`-- 'shared.cpp'-z`), git matches +# a file that does not exist, and the walk answers EMPTY — which surfaces as "git unavailable / no history", +# not as a broken command. Measured on the seam-broken build: --owners=sharedFn exited 1 with exactly that +# sentence while the all-files form beside it answered normally. This arm is the pin. +echo "arm c2: the single-file pathspec seam — --owners=SYM must reach its file" +"$BIN" --owners=sharedFn --detail=1 --no-cache "$G" > "$TMP/owners-sym.xml" 2>"$TMP/owners-sym.err"; sym_rc=$? +[ "$sym_rc" = 0 ] && ok "--owners=sharedFn exits 0 (the seam-broken build exited 1: 'git unavailable / no history')" \ + || no "--owners=sharedFn exits $sym_rc: $( head -1 "$TMP/owners-sym.err" )" +grep -q 'git unavailable' "$TMP/owners-sym.err" \ + && no "--owners=SYM answered 'git unavailable / no history' — the pathspec never reached git" \ + || ok "no 'git unavailable' on the SYM form" +grep -q ']*of="sharedFn"' "$TMP/owners-sym.xml" \ + && ok 'the SYM form echoes its selector in of=' \ + || no 'the SYM form does not echo of="sharedFn"' +sym_p=$( tr '>' '\n' < "$TMP/owners-sym.xml" | grep -oE ' "$TMP/golden.new" +if cmp -s "$GOLD" "$TMP/golden.new"; then + ok "owners+hotspots+cochange byte-identical to test/gitframingcheck_golden.txt (root= normalized)" +else + no "honest-tree drift from the golden:"; diff "$GOLD" "$TMP/golden.new" | head -10 +fi + +# ── ARM (e): the marker literal lives ONLY inside the shared helper ──────────────────────────────── +echo "arm e: the marker literal lives ONLY inside the shared helper" +code_hits=$( code_only "$SRC" | grep -o '__C__' | wc -l | tr -d ' ' ) +[ "$code_hits" = 2 ] \ + && ok "exactly 2 non-comment __C__ occurrences in src/gitmine.h" \ + || no "$code_hits non-comment __C__ occurrences in src/gitmine.h, want 2" +const_line=$( code_only "$SRC" | grep -c 'kGitLogMarker = "__C__"' ) +fmt_line=$( code_only "$SRC" | grep -c 'format=%x00__C__%x00%ae%x00%at%x00%ct' ) +if [ "$const_line" = 1 ] && [ "$fmt_line" = 1 ]; then + ok "the two sites are the kGitLogMarker constant and the shared command's --format= — no consumer body carries the marker" +else + no "the marker's code sites moved: constant=$const_line format=$fmt_line" +fi +tformat_hits=$( code_only "$SRC" | grep -o 'tformat:' | wc -l | tr -d ' ' ) +[ "$tformat_hits" = 0 ] && ok "the old line-frame spellings (tformat:__C__…) are gone from the code" \ + || no "$tformat_hits tformat: marker(s) remain in code" + +# ── ARM 2: the raw stream is byte-pinned ─────────────────────────────────────────────────────────── +echo "arm 2: the raw stream the reader parses is byte-pinned" +# The reader's contract is a BYTE layout, so the layout itself is pinned: the exact command +# gitLogRecordsCmd builds (its prefix flags — --no-optional-locks, the fsmonitor/signature pins and the +# driver pins — govern the index and the worktree, never `log` output, so plain git reproduces it). +"$REAL_GIT" -c core.quotepath=false -C "$H" log --no-ext-diff --no-textconv -c \ + -z --name-only --format='%x00__C__%x00%ae%x00%at%x00%ct' 2>/dev/null > "$TMP/raw.z" +digest=$( python3 -c 'import hashlib,sys; print(hashlib.sha256(open(sys.argv[1],"rb").read()).hexdigest())' "$TMP/raw.z" ) +[ "$digest" = a823b914f6269ee4b50c23204c407c9e009940fa7d62d530d7316de418a33b3d ] \ + && ok "the stream is byte-identical to the pinned digest" \ + || no "raw-stream digest drifted: $digest" +# Fields that spell the marker EXACTLY (NUL-delimited, so a '\n'-prefixed first path does not count): +# 11 record openers + 2 paths — c1's mid-list `__C__` (not first in byte order, hence no '\n' prefix) +# and the collision merge's bare first path. +fields=$( python3 -c ' +import sys +d = open( sys.argv[1], "rb" ).read() +needle = b"\x00__C__\x00" +n = 0 +i = d.find( needle ) +while i >= 0: + n += 1 + i = d.find( needle, i + 1 ) +print( n )' "$TMP/raw.z" ) +[ "$fields" = 13 ] \ + && ok "13 NUL-delimited fields spell the marker: 11 record openers + 2 marker-spelled paths" \ + || no "marker-spelled fields: $fields, want 13" + +# ── ARM 3: the merge shapes — the lookahead is load-bearing, the collision is pinned ──────────────── +echo "arm 3: the merge shapes" +alpha_churn=$( churn_of_fn "$TMP/for-alphaFn.xml" alphaFn ) +[ "$alpha_churn" = 4 ] \ + && ok "M2's name-bearing merge ATTACHES Alpha.cpp (churn=4; a parser without the one-field lookahead reads the merge's prefix empty as a boundary and drops its files — 3)" \ + || no "alphaFn churn='$alpha_churn', want 4 — the lookahead is not attaching merge paths" +dot_churn=$( churn_of_fn "$TMP/for-dotFn.xml" dotFn ) +[ "$dot_churn" = 3 ] \ + && ok "the disclosed collision is pinned: the collision merge's __C__.cpp is dropped (churn=3, not 4)" \ + || no "dotFn churn='$dot_churn', want 3 — the collision's observable moved" +hs_dot=$( attr_of "$TMP/hotspots.xml" '__C__.cpp' churn ) +[ "$hs_dot" = 4 ] \ + && ok "the two miners disagree there BY DESIGN: --hotspots rides the unmigrated gitCommandLines miner and still counts the collision merge's file (4)" \ + || no "--hotspots __C__.cpp churn='$hs_dot', want 4 — the documented disagreement moved, re-derive" + +# ── ARM 4: ownership answers with the AUTHOR, never the committer ────────────────────────────────── +echo "arm 4: author-not-committer" +zeta_top=$( attr_of "$TMP/owners.xml" 'zeta.cpp' top ) +[ "$zeta_top" = "real@x.com" ] \ + && ok "zeta.cpp is owned by its author real@x.com — c2's committer was Committer and the merges' Mergecommitter " \ + || no "zeta.cpp is owned by '$zeta_top'" +committers=$( grep -oE 'committer@x\.com|mergecommitter@x\.com' "$TMP/owners.xml" | wc -l | tr -d ' ' ) +[ "$committers" = 0 ] && ok "no committer email appears as an owner anywhere" \ + || no "$committers committer-email leak(s) in the owners output" +zeta_authors=$( attr_of "$TMP/owners.xml" 'zeta.cpp' authors ) +[ "$zeta_authors" = 2 ] \ + && ok "zeta.cpp has exactly 2 authors (real + quiet2; the committers of c2 and the merges do not count)" \ + || no "zeta.cpp authors='$zeta_authors', want 2" + +# ── ARM 5: non-vacuity — the fixture is what the arms claim ──────────────────────────────────────── +echo "arm 5: non-vacuity — the fixture is what the arms claim" +tracked=$( git -C "$H" ls-files | wc -l | tr -d ' ' ) +[ "$tracked" = 12 ] \ + && ok "12 tracked files: the 11 indexed plus the extension-less __C__ (unindexed BY DESIGN, layout-pin Finding 1)" \ + || no "tracked files: $tracked, want 12" +marker_tracked=$( git -C "$H" ls-files | grep -cx '__C__' ) +[ "$marker_tracked" = 1 ] && ok "a file named exactly __C__ is tracked — the name the frame must survive" \ + || no "the marker-named file is not tracked ($marker_tracked)" +files_n=$( grep -oE 'files="[0-9]+"' "$TMP/owners.xml" | head -1 | grep -oE '[0-9]+' ) +[ "$files_n" = 11 ] && ok "--owners analysed 11 files (files=11)" || no "owners files=$files_n, want 11" +ranked=$( grep -oE 'ranked="[0-9]+"' "$TMP/hotspots.xml" | head -1 | grep -oE '[0-9]+' ) +unranked=$( grep -oE 'unranked_no_complexity="[0-9]+"' "$TMP/hotspots.xml" | head -1 | grep -oE '[0-9]+' ) +[ "$ranked" = 11 ] && [ "$unranked" = 0 ] \ + && ok "--hotspots ranked all 11 files (every fixture file carries a branch, so nothing is unranked_no_complexity)" \ + || no "--hotspots ranked=$ranked unranked_no_complexity=$unranked, want 11/0" + +# ── ARM 6: determinism ───────────────────────────────────────────────────────────────────────────── +echo "arm 6: determinism — the reader's answers do not move between runs" +"$BIN" --owners --detail=1 --limit=60 "$H" > "$TMP/owners2.xml" 2>/dev/null +"$BIN" --hotspots --limit=60 "$H" > "$TMP/hotspots2.xml" 2>/dev/null +if cmp -s "$TMP/owners.xml" "$TMP/owners2.xml" && cmp -s "$TMP/hotspots.xml" "$TMP/hotspots2.xml"; then + ok "two runs byte-identical (owners + hotspots)" +else + no "the answers moved between runs" +fi + +# ── ARM 7: the oracle — the tool's number re-derived from git's own stream ───────────────────────── +echo "arm 7: the oracle" +# Trap #12: a number the tool agrees with itself about is not evidence. Re-derive Alpha.cpp's churn +# straight from git's own --name-only stream: every combined diff that lists it. (tr turns each NUL +# into a line end, which also splits git's '\n'-prefixed first paths — exactly the fields counted.) +oracle=$( "$REAL_GIT" -c core.quotepath=false -C "$H" log --no-ext-diff --no-textconv -c \ + -z --name-only --format='%x00__C__%x00' 2>/dev/null | tr '\0' '\n' | grep -cx 'Alpha.cpp' ) +[ "$oracle" = "$alpha_churn" ] \ + && ok "git's own stream lists Alpha.cpp $oracle times = the --for churn of $alpha_churn" \ + || no "oracle=$oracle, tool=$alpha_churn — the tool's number does not re-derive from git" + +# ── ARM 8: the git floor, and a git that cannot frame degrades honestly ──────────────────────────── +echo "arm 8: the git floor, and a git that rejects -z degrades honestly" +git_ver=$( git --version | grep -oE '[0-9]+\.[0-9]+(\.[0-9]+)?' | head -1 ) +gmaj=${git_ver%%.*}; gmin=${git_ver#*.}; gmin=${gmin%%.*} +if [ "$gmaj" -gt 2 ] || { [ "$gmaj" -eq 2 ] && [ "$gmin" -ge 43 ]; }; then + ok "git $git_ver >= 2.43.0 — the layout-pin floor (the '\n' separator, the merge prefix empty and the end-of-stream unterminated field are empirical; below the floor the pinned layout is unverified)" +else + no "git $git_ver is below the 2.43.0 floor — the pinned layout is unverified on this git" +fi +# A git that refuses -z: the reader must degrade HONESTLY — no fabricated rows, no silent half-answer. +SHIM="$TMP/shim"; mkdir -p "$SHIM" +cat > "$SHIM/git" <<'SHIM' +#!/usr/bin/env bash +for a in "$@"; do + if [ "$a" = "-z" ]; then + echo "git: the -z flag is not supported here" >&2 + exit 129 + fi +done +exec "$REAL_GIT" "$@" +SHIM +chmod +x "$SHIM/git" +PATH="$SHIM:$PATH" "$BIN" --owners --detail=1 --limit=60 "$H" > "$TMP/shim.xml" 2>"$TMP/shim.err"; shim_rc=$? +[ "$shim_rc" != 0 ] && ok "a git that rejects -z: --owners exits $shim_rc (non-zero)" \ + || no "a git that rejects -z: --owners exited 0 — a silent half-answer" +shim_rows=$( tr '>' '\n' < "$TMP/shim.xml" | grep -c '; bf=1 = one person holds it. window: shown= total= capped= has_more= next_offset= offset= limit= (capped=1 cut; next_offset= pastes as offset=). at=: commit+dirty+shallow. root=: p= relative to it. files=N: files analysed. --> +===== hotspots + +===== cochange + diff --git a/test/regression.sh b/test/regression.sh index 40bff09ee..505fc46ea 100755 --- a/test/regression.sh +++ b/test/regression.sh @@ -278,7 +278,7 @@ else fi # retired: forspinecheck — the --for caller spine (Change A) was dropped by its own offline kill rule: a spine of up to 2 caller hops from the top 3 seeds named a missed rubric item on 1 of 7 in-sample rows, so the gate that pinned it went with it # retired: cacheexclkeycheck — the per-configuration auto-cache key it pinned is a registered NEGATIVE (docs/EVALS.md, "The auto-cache key ignores --exclude", RUN 2026-09-03: a 158K-file root with >= 12 gate configurations thrashed the 2 GiB sweep); the retry design keeps ONE superset blob per root and will bring its own gate -for _g in a9disclosurecheck abicheck accessshapecheck ackonlycheck adaptivecheck adaptivecutshapecheck affectedcheck agentloopclaudecheck agentloopcodexcheck agentloopeditsuitecheck agentloopfollowupcheck agentloopgradercheck agentlooplockcheck agentloopopencodecheck agentsurfacecheck agenttablecheck aiderbytescheck anchorbodycheck anchorcheck archcheck archmetricscheck argvdiffcheck arisefollowupcheck ariseshimcheck aritycheck artifactcheck astqueryregexcheck astrocheck atcheck atomscheck attribmultidefcheck attrvocabcheck baselinecheck baselinedirtycheck baselineportcheck bashsourcecheck batchcheck binoverridecheck blindspotcheck bm25boundcheck bm25check bodiesshowncheck bodydialectcheck budgetpolicycheck buildtypestampcheck builtinbindcheck bundleidcheck cachefuzzcheck cachehashcheck cacheidentitycheck cacheisolationcheck cachelintcheck cacheoffsetcheck cachereservecheck cachesplitcheck callerscheck callformcheck callsrankordercheck candheadcheck candidatescheck canoncheck capdisclosurecheck capsweepcheck ccheck ccjsoncheck ceilingverdictcheck chacheck chaconecheck chainguardcheck chainidcheck childwalkscalecheck churndecaycheck churnjoincheck churnjsonstampcheck claudeconfigdircheck clicheck clojurecheck clonebandcheck clonecachecheck clonededupcheck cloneidiomcheck clonelexcheck clsrecvcheck cochangeboostcheck cochangecliocheck cochangesurprisecheck codexinstallhonestycheck codexplugincheck codexwrapcheck collectioncapcheck columnarattrcheck columnarcheck columnarcommacheck commentcoherencecheck commenttokencheck communitydrillcheck communitylabelcheck compactlegendcheck compactroutecheck completecheck composelangcheck connectcheck connectcorecheck connectjoincheck constcheck contextratiocheck coplintcheck countfloorcheck cppbenchcheck cppoperatorcheck cppqualcheck cpptmplscopecheck crashsweepcheck crawlescapecheck crossdirincludecheck crossrefcheck crossrefdegradecheck csharpcheck csharpcondcheck cudacheck cyclecutcheck danglingwarncheck dartcheck deadcheck deadfiltercheck deadprecisioncheck deckcheck deckclaimcheck declinecheck declinedlistcheck declinedsitecheck decltodefcheck deeptailcheck defaultceilingcheck defectshapecheck defoverdeclcheck degradedhintcheck dependencypincheck deplangscheck depsprecisecheck detailcheck diagnoticecheck didyoumeancheck dispatchordercheck dmmcheck docanchorcheck docdemotecheck docdriftcheck docdriftcommentcheck docmdcachecheck docmentioncheck docscommandscheck doctorcheck donelegendcheck droppedpositivecheck duprowcheck dynmapsimdcheck editcheckanswercheck editcheckcheck editcheckdeclcheck editchecknotecheck edithandlehintcheck editpayloadbinarycheck editplancheck editplanpayloadconfinecheck editplanrecheckcheck editplanrollbackmsgcheck editpreviewcheck editroundtripcheck edittargetfileabscheck eliximportcheck elixircheck elixirnamearitycheck elixirsemanticcheck emitescapecheck emittertruthcheck emptycorpuscheck emptyvaluerefusecheck ensembleavailcheck ensemblecheck enumtablecheck essentialcxcheck estchargecheck evalcheck evictioncheck exemplarcheck exemplarconfcheck exercisescheck expandbodyfirstcheck expandcallscheck expandmodecheck expandrangecheck expandsibscheck expandtokencheck expandtopk0check extentcheck externalvetocheck falseedgecheck fcallforwardcheck fficheck fieldaffinitycheck fieldidcheck fieldnarrowcheck fieldusescheck filerootcheck fileselectorrefusecheck fillordercheck fixedbufsweep flagscheck flagsnoisecheck flagsurfacecheck flagtablecheck flipcheck floormarkcheck fnliteralcheck fnptrcheck forautobodycheck forblowupcheck forbodyfoldcheck forbodyslotcheck forbudgetmonotoncheck forcalibfactscheck forcompresscheck fordisclosurecheck forhdrshapecheck forheadrrcheck forlenscheck formatgatecheck formaxtokenscheck formembernamescheck fornotesbudgetcheck fornotesjsoncheck forrankordercheck forrootlegendcheck forsectioncollapsecheck forsigspancheck forsoftbudgetcheck forwidencheck freshclonecheck freshnesscheck g1configcheck gateabilitycheck gatecountcheck gateexitcheck gdscriptcheck genrecallcheck gitchildoptscheck gitenvhermeticcheck githardencheck gitignorecheck gitquotepathcheck gitstampcheck goinstcheck gointerfacecheck gokindcheck goscopecheck graphlegendbudgetcheck graphqueryrefusecheck grepanchorcheck grepandcheck grepbytescheck grepcheck grepcontextcheck grepcorpuscheck grepfastcheck grepfollowupcheck grepignorecheck grepscancheck grepseamcheck greptiercheck guardmsgcheck hasacheck hazardpatterncheck headbinstagecheck headsnapcachecheck helpbudgetcheck hermesinstallcheck historyoraclecheck hookcheck hostilecheck hotspotsincecheck htmlcolorcheck htmlhostcheck htmlrendercheck identitycheck impactdepthcheck impactimportcheck impactpartitioncheck impacttypecheck importcapcheck importnarrowcheck includeanglecheck includeprecisecheck indexoutcheck infraportcheck isolateprovenancecheck javamethodrefcheck javarubycheck jslangcheck jsmetricscheck jsnestedcheck jsoncheck jsonlangcheck jsonparitycheck jsonredactcheck jsonrefusallegendcheck jsonwalkcheck jsshapecheck jsverbscheck jsxcallcheck knownitemcheck kotlincheck landingcheck langcensuscheck langcheck langidentitycheck layerquerycheck layoutcheck lb3namecheck legendcostcheck legendcoveragecheck legenddriftcheck legendrefcheck legobundlecheck legocheck legotransitivecheck liftdisclosurecheck limitstablecheck lintbudgetcheck lintcatalogcheck lintcheck lintdedupcheck lintpayloadcapcheck lintprecisioncheck lintrulescheck lintscopecheck lintselectcheck listingpagingcheck localdeclcheck localitycheck localscountcheck loopconservationcheck lpincheck luacheck luarequirecheck macroedgecheck macroreparsecheck manifestcheck mapdatasectioncheck mapdiffcheck mapinventorycheck matchcapturecheck matchgrammarcheck maxfilesizecheck mcpattrparitycheck mcpaudit4hardencheck mcpclidiffcheck mcpcodexmetacheck mcpcontractcheck mcpdegradedhintcheck mcpeditcheck mcpeditkindcheck mcpeditmodecheck mcpeditpresencecheck mcpeditracecheck mcpflagshipcheck mcpforparitycheck mcpframehonestycheck mcpgrepdegradedcheck mcphandlecheck mcpincrementalcheck mcpmanifestcheck mcprangeedgecheck mcpreadloopcheck mcpredactcheck mcpreloadasancheck mcpreloadcheck mcpremotecheck mcprobustcheck mcpslicecheck mcpstalecheck mcpstdiolinecapcheck mcpstrictschemacheck mcptoolprunecheck mcptoolsubsetcheck mcptranchecheck mcptwinclaimscheck mcpverbscheck mcpw2fixcheck mcpw3fixcheck mcpwatchercheck mdembedcheck mdsectioncheck memguardcheck mentioncapcheck mentioncheck mentionsverbcheck mergechurncheck mergescoutcheck mergescoutlonglinecheck metalcheck meterdisclosurecheck metricscheck modifierguardcheck moduleconstcheck morecontractcheck mrowalkcheck multirootcheck multiswecheck namedfileinputcheck nameinfocheck namingcalibrationcheck namingconsistencycheck naminglenscheck naminglocalscheck narrowcheck narrowlangcheck neighbourcapcheck nestedimportcheck nestedqualcheck nestprofilecheck nextverbcheck noaliascheck nodekindcheck nongitqmetricscheck nonlocalstatecheck notecanoncheck notescheck notesdegradecheck nsfiltercheck nulbytecheck numericrefusecheck objcfieldcheck objcsniffcheck opencodewrapcheck optremarkscheck optremarkshotcheck ordercheck osswitchcheck oswin32logiccheck outlinecheck overbudgetcommentcheck ownerhopcheck ownerscheck packcallersharecheck packtaskcheck packtaskmonotoncheck packtaskquotacheck padscalecheck paginationcheck pagingsweepcheck panellegendcheck pargatescheck parsehealthcheck partitioncheck pathgapcheck pathownercheck patterncheck perfharnesscheck phpcheck pincensuscheck planlanescheck planlintcheck pmccheck portablebuildcheck portablecachecheck postingscheck ppaltcheck ppdeadrolescheck pranchorcheck prbudgetcheck prcheck prcontextcheck prconvergecheck precedencecheck preproccondcheck preprocdeadscalecheck prmaskanchorcheck prnestedcapcheck probecheck propcostcheck prrefsafecheck prrenamecheck pyimportprecisecheck pymodulealiascheck pyshapecheck qackconcurrencycheck qackorigincheck qbaselineproducercheck qchurncheck qchurnmemocheck qddegradecheck qddialscheck qdrefpaircheck qextractionkeycheck qoriginoraclecheck qrevtokencheck qrowlocatorcheck qschemetripcheck qsnapcachecheck qsnapprefetchcheck qsnapproducercheck qualifiedresolvecheck qualitycheck qualitycrosslangcheck qualityexcludecheck qualitykeycheck qualitykindscheck qualityorigincheck qualitypanelcheck qualityscopecheck qualitysignalcheck qualitystalecheck qualitysymcheck qualnewcheck querycheck queryfilescancheck racymtimecheck radixsimdcheck rangecomposecheck rankbycheck reachcheck readabilitycheck readmedriftcheck readmeexamplecheck recallanchorcheck recallboundarycheck recallbudgetcheck recallbufcheck recallevalcheck recallparitycheck recallpassagecheck recallrankdepthcheck recallrelcheck recallshapecheck recalltablecheck recalltotalcheck receiptpostcheck receiverevidencecheck recentscopecheck redactcheck redactfixcheck refusaltailcheck regexbombcheck regexcheck regexguardcheck regexrefusecheck registermacrocheck relevancefloorcheck relinkcheck reportcheck resolvecheck resolverhonestycheck retrievalqualitycheck reusefirstworkflowcheck ripwirepubliccheck rootrelcheck rootrelemitcheck rootspellingcheck routecheck routeedgecheck routehookcheck routeoncecheck routingreportcheck rubyargcheck rubyattrscheck rubybarecallcheck rubyclassrecvcheck rubyconstcheck rubydeclrefcheck rubydescribedclasscheck rubyinheritcheck rubymetricscheck rubyrakejbuildercheck rubyreachcheck rubyrecvcheck rubyrecvnarrowcheck rubyrequirecheck rubyrspectargetcheck rubyschemacheck rubyscopecheck rubysettercheck rubytypedrecvcheck runhintcheck runtracecheck rustanccheck rustimportprecisecheck rustqualcheck safedeletecheck sarifcheck savecachecheck scipcheck scipjoincheck scorecardcheck scoutheadconflictcheck scoutkeycheck scroundtripcheck seedboundscheck selectorchaincheck selectorhonestycheck selectorrefusecheck selectorscopecheck selfcheckcheck selfcontainedcheck shadowcheck shallowhistorycheck shapingflagcheck shellgateindexcheck showcasecapturecheck sibliftcheck sidecarsymlinkcheck sigredactcheck sincecheck sincecochangecheck sincewindowcheck singledefcheck situdiffcheck situshapecheck skilldescbudgetcheck skillevalcheck skillevalsplitcheck skillinstallcheck skillroutingjudgedcheck skillscanreadcheck skilltruthcheck skipclassifycheck skippedcheck skipreasoncheck slicecheck slicediffcheck sliceflowcheck sliceflowsenscheck slotjoincheck spectimingcheck staleackcheck statgatecheck stdmembergatecheck stdqualcheck strkerncheck structlayoutcheck sublistcountcheck subscriptopcheck substrfiltercheck subtokencheck svectorcheck swiftcheck swiftmemberscheck swiftshapecheck taskechocheck tempfilesymlinkcheck termmargincheck testedreachcheck testgatecheck testgatelegendbudgetcheck testgatepagecheck testgaterefusecheck testmacrocheck testrowruncheck testscopecheck textdocscheck timsortcheck tokenbudgetcheck tomllangcheck toolcallroutecheck tornreadcheck traceasanlinearcheck tracecheck tracehandoffcapcheck tracehopcheck traceminecheck treecheck truncvocabcheck tsimportprecisecheck tsshapecheck type3check type3clonecheck typerefcheck unreachablecheck unresolvedcheck usescheck usesliteralcheck usesselectorcheck usingdeclcheck utf8scrubcheck vendoredassetcheck vendoredbundlecheck vendorpatchcheck verifycheck versioncheck w2verbscheck w3fixbudgetcheck w3fixlegendcheck weaksignalcheck withgraphcheck withprofilecheck worktreeleakcheck wrapverbscheck writetargetcheck xmlwellformed yamllangcheck zonecheck zoneconsistencycheck zoomcheck; do +for _g in a9disclosurecheck abicheck accessshapecheck ackonlycheck adaptivecheck adaptivecutshapecheck affectedcheck agentloopclaudecheck agentloopcodexcheck agentloopeditsuitecheck agentloopfollowupcheck agentloopgradercheck agentlooplockcheck agentloopopencodecheck agentsurfacecheck agenttablecheck aiderbytescheck anchorbodycheck anchorcheck archcheck archmetricscheck argvdiffcheck arisefollowupcheck ariseshimcheck aritycheck artifactcheck astqueryregexcheck astrocheck atcheck atomscheck attribmultidefcheck attrvocabcheck baselinecheck baselinedirtycheck baselineportcheck bashsourcecheck batchcheck binoverridecheck blindspotcheck bm25boundcheck bm25check bodiesshowncheck bodydialectcheck budgetpolicycheck buildtypestampcheck builtinbindcheck bundleidcheck cachefuzzcheck cachehashcheck cacheidentitycheck cacheisolationcheck cachelintcheck cacheoffsetcheck cachereservecheck cachesplitcheck callerscheck callformcheck callsrankordercheck candheadcheck candidatescheck canoncheck capdisclosurecheck capsweepcheck ccheck ccjsoncheck ceilingverdictcheck chacheck chaconecheck chainguardcheck chainidcheck childwalkscalecheck churndecaycheck churnjoincheck churnjsonstampcheck claudeconfigdircheck clicheck clojurecheck clonebandcheck clonecachecheck clonededupcheck cloneidiomcheck clonelexcheck clsrecvcheck cochangeboostcheck cochangecliocheck cochangesurprisecheck codexinstallhonestycheck codexplugincheck codexwrapcheck collectioncapcheck columnarattrcheck columnarcheck columnarcommacheck commentcoherencecheck commenttokencheck communitydrillcheck communitylabelcheck compactlegendcheck compactroutecheck completecheck composelangcheck connectcheck connectcorecheck connectjoincheck constcheck contextratiocheck coplintcheck countfloorcheck cppbenchcheck cppoperatorcheck cppqualcheck cpptmplscopecheck crashsweepcheck crawlescapecheck crossdirincludecheck crossrefcheck crossrefdegradecheck csharpcheck csharpcondcheck cudacheck cyclecutcheck danglingwarncheck dartcheck deadcheck deadfiltercheck deadprecisioncheck deckcheck deckclaimcheck declinecheck declinedlistcheck declinedsitecheck decltodefcheck deeptailcheck defaultceilingcheck defectshapecheck defoverdeclcheck degradedhintcheck dependencypincheck deplangscheck depsprecisecheck detailcheck diagnoticecheck didyoumeancheck dispatchordercheck dmmcheck docanchorcheck docdemotecheck docdriftcheck docdriftcommentcheck docmdcachecheck docmentioncheck docscommandscheck doctorcheck donelegendcheck droppedpositivecheck duprowcheck dynmapsimdcheck editcheckanswercheck editcheckcheck editcheckdeclcheck editchecknotecheck edithandlehintcheck editpayloadbinarycheck editplancheck editplanpayloadconfinecheck editplanrecheckcheck editplanrollbackmsgcheck editpreviewcheck editroundtripcheck edittargetfileabscheck eliximportcheck elixircheck elixirnamearitycheck elixirsemanticcheck emitescapecheck emittertruthcheck emptycorpuscheck emptyvaluerefusecheck ensembleavailcheck ensemblecheck enumtablecheck essentialcxcheck estchargecheck evalcheck evictioncheck exemplarcheck exemplarconfcheck exercisescheck expandbodyfirstcheck expandcallscheck expandmodecheck expandrangecheck expandsibscheck expandtokencheck expandtopk0check extentcheck externalvetocheck falseedgecheck fcallforwardcheck fficheck fieldaffinitycheck fieldidcheck fieldnarrowcheck fieldusescheck filerootcheck fileselectorrefusecheck fillordercheck fixedbufsweep flagscheck flagsnoisecheck flagsurfacecheck flagtablecheck flipcheck floormarkcheck fnliteralcheck fnptrcheck forautobodycheck forblowupcheck forbodyfoldcheck forbodyslotcheck forbudgetmonotoncheck forcalibfactscheck forcompresscheck fordisclosurecheck forhdrshapecheck forheadrrcheck forlenscheck formatgatecheck formaxtokenscheck formembernamescheck fornotesbudgetcheck fornotesjsoncheck forrankordercheck forrootlegendcheck forsectioncollapsecheck forsigspancheck forsoftbudgetcheck forwidencheck freshclonecheck freshnesscheck g1configcheck gateabilitycheck gatecountcheck gateexitcheck gdscriptcheck genrecallcheck gitchildoptscheck gitenvhermeticcheck gitframingcheck githardencheck gitignorecheck gitquotepathcheck gitstampcheck goinstcheck gointerfacecheck gokindcheck goscopecheck graphlegendbudgetcheck graphqueryrefusecheck grepanchorcheck grepandcheck grepbytescheck grepcheck grepcontextcheck grepcorpuscheck grepfastcheck grepfollowupcheck grepignorecheck grepscancheck grepseamcheck greptiercheck guardmsgcheck hasacheck hazardpatterncheck headbinstagecheck headsnapcachecheck helpbudgetcheck hermesinstallcheck historyoraclecheck hookcheck hostilecheck hotspotsincecheck htmlcolorcheck htmlhostcheck htmlrendercheck identitycheck impactdepthcheck impactimportcheck impactpartitioncheck impacttypecheck importcapcheck importnarrowcheck includeanglecheck includeprecisecheck indexoutcheck infraportcheck isolateprovenancecheck javamethodrefcheck javarubycheck jslangcheck jsmetricscheck jsnestedcheck jsoncheck jsonlangcheck jsonparitycheck jsonredactcheck jsonrefusallegendcheck jsonwalkcheck jsshapecheck jsverbscheck jsxcallcheck knownitemcheck kotlincheck landingcheck langcensuscheck langcheck langidentitycheck layerquerycheck layoutcheck lb3namecheck legendcostcheck legendcoveragecheck legenddriftcheck legendrefcheck legobundlecheck legocheck legotransitivecheck liftdisclosurecheck limitstablecheck lintbudgetcheck lintcatalogcheck lintcheck lintdedupcheck lintpayloadcapcheck lintprecisioncheck lintrulescheck lintscopecheck lintselectcheck listingpagingcheck localdeclcheck localitycheck localscountcheck loopconservationcheck lpincheck luacheck luarequirecheck macroedgecheck macroreparsecheck manifestcheck mapdatasectioncheck mapdiffcheck mapinventorycheck matchcapturecheck matchgrammarcheck maxfilesizecheck mcpattrparitycheck mcpaudit4hardencheck mcpclidiffcheck mcpcodexmetacheck mcpcontractcheck mcpdegradedhintcheck mcpeditcheck mcpeditkindcheck mcpeditmodecheck mcpeditpresencecheck mcpeditracecheck mcpflagshipcheck mcpforparitycheck mcpframehonestycheck mcpgrepdegradedcheck mcphandlecheck mcpincrementalcheck mcpmanifestcheck mcprangeedgecheck mcpreadloopcheck mcpredactcheck mcpreloadasancheck mcpreloadcheck mcpremotecheck mcprobustcheck mcpslicecheck mcpstalecheck mcpstdiolinecapcheck mcpstrictschemacheck mcptoolprunecheck mcptoolsubsetcheck mcptranchecheck mcptwinclaimscheck mcpverbscheck mcpw2fixcheck mcpw3fixcheck mcpwatchercheck mdembedcheck mdsectioncheck memguardcheck mentioncapcheck mentioncheck mentionsverbcheck mergechurncheck mergescoutcheck mergescoutlonglinecheck metalcheck meterdisclosurecheck metricscheck modifierguardcheck moduleconstcheck morecontractcheck mrowalkcheck multirootcheck multiswecheck namedfileinputcheck nameinfocheck namingcalibrationcheck namingconsistencycheck naminglenscheck naminglocalscheck narrowcheck narrowlangcheck neighbourcapcheck nestedimportcheck nestedqualcheck nestprofilecheck nextverbcheck noaliascheck nodekindcheck nongitqmetricscheck nonlocalstatecheck notecanoncheck notescheck notesdegradecheck nsfiltercheck nulbytecheck numericrefusecheck objcfieldcheck objcsniffcheck opencodewrapcheck optremarkscheck optremarkshotcheck ordercheck osswitchcheck oswin32logiccheck outlinecheck overbudgetcommentcheck ownerhopcheck ownerscheck packcallersharecheck packtaskcheck packtaskmonotoncheck packtaskquotacheck padscalecheck paginationcheck pagingsweepcheck panellegendcheck pargatescheck parsehealthcheck partitioncheck pathgapcheck pathownercheck patterncheck perfharnesscheck phpcheck pincensuscheck planlanescheck planlintcheck pmccheck portablebuildcheck portablecachecheck postingscheck ppaltcheck ppdeadrolescheck pranchorcheck prbudgetcheck prcheck prcontextcheck prconvergecheck precedencecheck preproccondcheck preprocdeadscalecheck prmaskanchorcheck prnestedcapcheck probecheck propcostcheck prrefsafecheck prrenamecheck pyimportprecisecheck pymodulealiascheck pyshapecheck qackconcurrencycheck qackorigincheck qbaselineproducercheck qchurncheck qchurnmemocheck qddegradecheck qddialscheck qdrefpaircheck qextractionkeycheck qoriginoraclecheck qrevtokencheck qrowlocatorcheck qschemetripcheck qsnapcachecheck qsnapprefetchcheck qsnapproducercheck qualifiedresolvecheck qualitycheck qualitycrosslangcheck qualityexcludecheck qualitykeycheck qualitykindscheck qualityorigincheck qualitypanelcheck qualityscopecheck qualitysignalcheck qualitystalecheck qualitysymcheck qualnewcheck querycheck queryfilescancheck racymtimecheck radixsimdcheck rangecomposecheck rankbycheck reachcheck readabilitycheck readmedriftcheck readmeexamplecheck recallanchorcheck recallboundarycheck recallbudgetcheck recallbufcheck recallevalcheck recallparitycheck recallpassagecheck recallrankdepthcheck recallrelcheck recallshapecheck recalltablecheck recalltotalcheck receiptpostcheck receiverevidencecheck recentscopecheck redactcheck redactfixcheck refusaltailcheck regexbombcheck regexcheck regexguardcheck regexrefusecheck registermacrocheck relevancefloorcheck relinkcheck reportcheck resolvecheck resolverhonestycheck retrievalqualitycheck reusefirstworkflowcheck ripwirepubliccheck rootrelcheck rootrelemitcheck rootspellingcheck routecheck routeedgecheck routehookcheck routeoncecheck routingreportcheck rubyargcheck rubyattrscheck rubybarecallcheck rubyclassrecvcheck rubyconstcheck rubydeclrefcheck rubydescribedclasscheck rubyinheritcheck rubymetricscheck rubyrakejbuildercheck rubyreachcheck rubyrecvcheck rubyrecvnarrowcheck rubyrequirecheck rubyrspectargetcheck rubyschemacheck rubyscopecheck rubysettercheck rubytypedrecvcheck runhintcheck runtracecheck rustanccheck rustimportprecisecheck rustqualcheck safedeletecheck sarifcheck savecachecheck scipcheck scipjoincheck scorecardcheck scoutheadconflictcheck scoutkeycheck scroundtripcheck seedboundscheck selectorchaincheck selectorhonestycheck selectorrefusecheck selectorscopecheck selfcheckcheck selfcontainedcheck shadowcheck shallowhistorycheck shapingflagcheck shellgateindexcheck showcasecapturecheck sibliftcheck sidecarsymlinkcheck sigredactcheck sincecheck sincecochangecheck sincewindowcheck singledefcheck situdiffcheck situshapecheck skilldescbudgetcheck skillevalcheck skillevalsplitcheck skillinstallcheck skillroutingjudgedcheck skillscanreadcheck skilltruthcheck skipclassifycheck skippedcheck skipreasoncheck slicecheck slicediffcheck sliceflowcheck sliceflowsenscheck slotjoincheck spectimingcheck staleackcheck statgatecheck stdmembergatecheck stdqualcheck strkerncheck structlayoutcheck sublistcountcheck subscriptopcheck substrfiltercheck subtokencheck svectorcheck swiftcheck swiftmemberscheck swiftshapecheck taskechocheck tempfilesymlinkcheck termmargincheck testedreachcheck testgatecheck testgatelegendbudgetcheck testgatepagecheck testgaterefusecheck testmacrocheck testrowruncheck testscopecheck textdocscheck timsortcheck tokenbudgetcheck tomllangcheck toolcallroutecheck tornreadcheck traceasanlinearcheck tracecheck tracehandoffcapcheck tracehopcheck traceminecheck treecheck truncvocabcheck tsimportprecisecheck tsshapecheck type3check type3clonecheck typerefcheck unreachablecheck unresolvedcheck usescheck usesliteralcheck usesselectorcheck usingdeclcheck utf8scrubcheck vendoredassetcheck vendoredbundlecheck vendorpatchcheck verifycheck versioncheck w2verbscheck w3fixbudgetcheck w3fixlegendcheck weaksignalcheck withgraphcheck withprofilecheck worktreeleakcheck wrapverbscheck writetargetcheck xmlwellformed yamllangcheck zonecheck zoneconsistencycheck zoomcheck; do [ -f "$ROOT/test/$_g.sh" ] || continue if RIPWIRE_BIN="$BIN" bash "$ROOT/test/$_g.sh" >/dev/null 2>&1; then ok "absorb gate ($_g.sh)" From fa474048417e95125f9246fdfcafdab6362e7b8b Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 18:33:12 +0100 Subject: [PATCH 05/15] git-porcelain-framing: the dead code goes, and the churn memo's scheme moves to 4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three pieces of finishing work on the src side of the porcelain-framing lane. The migration left one artifact behind: gitLogRecords, the collecting form of forEachGitLogRecord that kept every record in a vector "for the one consumer that keeps every record". There is no such consumer — the five readers are all visitors now, and nothing in the tree calls it. It goes, and with it the GitLogWalk::records member it alone used: the walk's contract is its exit status (unread/notRun/exitCode — the K51 "a prefix is not an answer" rule, which the shared reader now applies itself), and the struct's own comment already said so. What is left is exactly what the streaming design needs: one record in memory at a time. The same migration orphaned the two helpers that used to carry that contract into the raw stream: markWalkUnread and noteWalkExit, whose last callers lived in the old collecting walk (baseline src/gitmine.h:1614-1661). The shared reader now applies the rule itself, and every consumer reads unread/notRun/exitCode straight off the walk — so nothing calls them. A helper that lost its last caller is dead code, and on this tree that is a GATING quality finding: --quality-delta reports it preexisting-worse (the symbols were live at baseline), and preexisting-worse gates when major. Leaving them defined to be tidied up later is precisely what the gate exists to refuse, so they go here, in the commit that took their callers away. And the cache that holds the walk's output moves with it: kQChurnCacheScheme 3 -> 4. On a repository whose committed paths spell the marker bytes, carry a newline, or sort after a path that does, the OLD reader's stream was corrupted, so a blob written by a pre-lane binary would keep serving that corrupted stream to a fixed one until HEAD moved — the same wrong-answer-from-cache as scheme 2 (the merge-blind command), one layer down: there the COMMAND changed under an unchanged key, here the PARSE did. Neither is visible to the key, which is why this bump is manual. On ordinary trees the two readers' streams are byte-identical (the lane's golden arm), so the bump costs one cold walk per repository and changes no answer. Verification: --quality-delta on this tree reports gating="0" — the lane adds no blocking quality findings. gitframingcheck, mergechurncheck, churnjoincheck, qchurncheck, qchurnmemocheck, qschemetripcheck and evictioncheck all pass on the rebuilt binary. --- src/gitmine.h | 33 ++------------------------------- src/quality.h | 11 ++++++++++- 2 files changed, 12 insertions(+), 32 deletions(-) diff --git a/src/gitmine.h b/src/gitmine.h index ca04656db..ceb869862 100644 --- a/src/gitmine.h +++ b/src/gitmine.h @@ -775,9 +775,10 @@ struct GitLogRecord // The walk's exit status, carried so the K51 contract (a prefix is not an answer) survives the // migration: `unread` — git would not start, or died part-way, so the records are empty or a prefix; // `notRun` — git never ran at all (no popen, not on PATH, not executable), so there is no prefix either. +// The visitor form streams: there is no records vector to keep, which is the point — one record in +// memory at a time, the whole-history walk costs what it always cost. struct GitLogWalk { - std::vector records; bool unread = false; bool notRun = false; int exitCode = 0; @@ -938,18 +939,6 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin return walk; } -// The collecting form, for the one consumer that keeps every record: the raw stream the churn / -// co-change core is resolved against, memoized per (repo, HEAD, window). -inline GitLogWalk gitLogRecords( const std::string& root, const std::string& extraArgs ) -{ - GitLogWalk walk; - const GitLogWalk streamed = forEachGitLogRecord( root, extraArgs, [ & ]( const GitLogRecord& r ) { walk.records.push_back( r ); } ); - walk.unread = streamed.unread; - walk.notRun = streamed.notRun; - walk.exitCode = streamed.exitCode; - return walk; -} - // The repo TOPLEVEL enclosing `absDir` (absolute and already realpath'd), or empty when `absDir` is not inside // a git repo at all. realpath'd on the way out so it compares byte-for-byte with a realpath'd file path — on // this platform `/tmp` is a symlink to `/private/tmp`, and an unresolved toplevel would fail every prefix test @@ -1795,24 +1784,6 @@ struct RawCommitStream int exitCode = 0; }; -// K51: mark a walk unread. `exitCode` is git's decoded exit (-1: it never started, or did not exit normally); the shell's 127 -// (not on PATH) and 126 (not executable) mean git never ran either, so `commits` is empty, not a prefix. -inline void markWalkUnread( RawCommitStream& out, int exitCode, bool neverStarted ) noexcept -{ - out.unread = true; - out.exitCode = exitCode; - out.notRun = neverStarted || exitCode == 127 || exitCode == 126; -} - -// K51: fold the walk's pclose() status into `out` — 0 is a clean read and changes nothing. -inline void noteWalkExit( RawCommitStream& out, int status ) noexcept -{ - if( status != 0 ) - { - markWalkUnread( out, decodedExitCode( status ), false ); - } -} - inline RawCommitStream gitLogNameOnlyRaw( const std::string& root, const std::string& coSince ) { PROFILE_SCOPE_DESCRIBE( "gitmine: gitLogNameOnlyRaw (git log --name-only popen)" ); diff --git a/src/quality.h b/src/quality.h index d927d6dc9..542200856 100644 --- a/src/quality.h +++ b/src/quality.h @@ -4939,7 +4939,16 @@ computeWindowRefBodyHashes( const std::string& root, std::uint32_t days, // newest prefix of the window) as if it were the window, and the key cannot tell that blob from a real read. K51 never // stores one, but a blob already on disk would keep serving its silent zero or prefix, with no history_unread=, until HEAD // moved. The bump makes every pre-K51 blob a clean miss; the cost is one cold walk per repository. -constexpr std::uint32_t kQChurnCacheScheme = 3; +// SCHEME 4 (2026-10-23, the git-porcelain-framing lane): the walk's READER was rebuilt — the line-framed +// `__C__` marker parse became one NUL-framed reader with a whole-field sentinel and one field of lookahead +// (test/gitframingcheck.sh). On a repository whose committed paths spell the marker bytes, carry a +// newline, or sort after a path that does, the OLD reader's stream was corrupted (forged commit headers, +// swallowed paths), so a blob written by a pre-lane binary would keep serving that corrupted stream to a +// fixed one until HEAD moved — the same wrong-answer-from-cache as scheme 2, one layer down: there the +// COMMAND changed under an unchanged key, here the PARSE did. Neither is visible to the key, which is why +// this bump is manual. On ordinary trees the two readers' streams are byte-identical (the lane's golden +// arm over owners/hotspots/cochange), so the bump costs those repos one cold walk and changes no answer. +constexpr std::uint32_t kQChurnCacheScheme = 4; constexpr char kQChurnMagic[4] = { 'Q', 'C', 'H', 'N' }; inline std::string serializeRawCommitStream( const RawCommitStream& raw, const std::string& keyMat ) From 26f258975d951b2d0417b50467769b887fa3e163 Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 18:51:01 +0100 Subject: [PATCH 06/15] =?UTF-8?q?docs:=20CHANGELOG=20[Unreleased]=20?= =?UTF-8?q?=E2=80=94=20the=20git-log=20framing=20fix,=20with=20its=20cache?= =?UTF-8?q?=20consequence?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lane's user-facing entry: what the defect was (a committed filename read as commit metadata, with the measured numbers from the hostile fixture), what the one shared NUL-framed reader changes for every POSIX-legal filename, the golden that proves ordinary trees are untouched, and the one case disclosed rather than fixed (a merge whose first combined path is the marker-spelled file), whose full fix is a follow-on spec's format sentinel change. The kQChurnCacheScheme 3 -> 4 move is stated here too, in the same sentence shape 0.6.8 used for its own 2 -> 3: what a pre-lane blob would keep serving, and what the upgrade costs. --- CHANGELOG.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b244c95e..dc1f4353a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -147,6 +147,37 @@ plain and its one cold whole-repo call (22-30 s under ASan) timed out twice in n from the runtime's symbol names inside the binary or from the runtime answering `ASAN_OPTIONS=help=1`; the alarms themselves, every arm and every literal are unchanged. Test-only; no output change. +### Fixed — a committed filename can no longer be read as commit metadata (`git log` framing) + +Every `git log --name-only` consumer in the history miners framed its stream with a line-based reader and a +`__C__` marker matched as a line prefix, so a repository containing a file whose name begins with the marker +bytes had that file read as commit metadata. Measured on a hostile fixture (files named `__C__`, +`__C__x|1694000000.cpp`, `__C__ 1694000000.cpp`, `ab.cpp`, `q"uote.cpp`, `sp ace.cpp`, `back\slash.cpp`, +`pi|pe.cpp`, `--dashdash.cpp`): `--owners` analysed 8 of 11 files, five of them owned by an author email that +appears in no commit, one file's top owner flipped to a different real author, and the newline-named file lost +its churn entirely. + +One reader now parses the stream for all five consumers (`gitLogFileSets`, `gitLogNameOnlyRaw`, +`gitFileCommitCountsInDayWindow`, `gitLogDecayedFileMining`, `gitFileAuthors`): +`git log -z --name-only --format=%x00__C__%x00%ae%x00%at%x00%ct`, with the marker a whole-field sentinel a path +can never spell and one field of lookahead so a merge's prefix empty is not mistaken for a record boundary. +Any POSIX-legal filename — newlines, quotes, backslashes, pipes, leading dashes, the marker itself — +round-trips byte-exact, and ownership answers with the author, never the committer. On trees with ordinary +names the answers are byte-identical to before, pinned by a golden recorded from the pre-change binary over +`--owners`, `--hotspots` and `--cochange`. + +The history-walk cache's `kQChurnCacheScheme` moves 3 → 4: a blob written by a pre-lane binary holds the old +reader's stream, which for the repositories this fix targets is the corrupted one, and no key component can +tell that blob from a fresh read. Every repository's history is walked once more after the upgrade; on trees +with ordinary names the walk produces the same stream, so the cost is one cold walk and no changed answer. + +**One case is disclosed, not fixed:** a merge whose FIRST combined path is a file literally named `__C__` +still reads as pathless — its prefix empty is followed by the marker-spelled path, which the boundary rule +cannot tell from a real record boundary, so that merge's files lose their attribution (a pathless phantom +record is emitted that every consumer ignores). The full fix is a format sentinel change, owned by a +follow-on spec. `test/gitframingcheck.sh` pins both the fix and this collision's exact observable, including +where the two churn miners disagree by design. + ### Changed — the versions this change moves, stated once `kParserVer` 206 → 220 (file tag 221): the Rust written-type receivers carried 202 (203) on their branch and the struct-field From 26361fbdc6227389cd588ca5e0983bcb47d0379a Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 23:59:44 +0100 Subject: [PATCH 07/15] fix(gitmine): the lookahead hand-over is a copy, and the dead ternary goes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #433's style leg (clang-tidy 22, bugprone-use-after-move) reads the walk loop's hand-over of the held lookahead as a second use of a moved-from object: the slot is moved at the top of one iteration and moved again in a later one, with the reassignment that makes that safe sitting between them where no analyzer can prove the correlation with the haveLookahead flag. A swap is no better — it reads the slot too, so the checker relocates the finding to the path append below. The fix is a COPY: the slot is a reusable buffer, the two writers each assign a FRESH value before they set the flag, and no read ever sees a moved-from slot. It runs once per record boundary — and once per a merge's prefix empty — against a walk that spawns a git child per run, so it costs nothing measurable. The same loop carried a `haveLookahead ? … : fields.next( next )` at the empty-field check that could never take its true branch: the top of the loop consumes the held field and clears the flag before any field is examined. Removed, with the invariant stated in a comment so it is not "restored" — defensive code for an invariant the loop does not have is how the move got there. Behavior-preserving: the only difference is the slot's post-hand-over value, which no read can observe — the flag is cleared here and every reader of the slot is gated on it. The framing gate runs byte-identical on the finished state (same 35 arms, same digest pin, same golden). A bisecter reaching this commit is green on the style leg and still red on memguardcheck's B21 until the next commit lands the abandon hook. --- src/gitmine.h | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/gitmine.h b/src/gitmine.h index ceb869862..8ce573f19 100644 --- a/src/gitmine.h +++ b/src/gitmine.h @@ -877,15 +877,31 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin while( !atEnd ) { if( gitWalksAbandoned() ) { break; } // the walk readers' abandon contract: stop at the next field; pclose ends the child - if( haveLookahead ) { field = std::move( lookahead ); haveLookahead = false; } + // COPY, not move: the slot is a reusable buffer, and a copy hands the held value over while + // leaving the slot in a valid state for its next fill. No read ever sees a moved-from slot — + // the two writers below each assign a FRESH value before they set the flag — but a move here + // is what bugprone-use-after-move reads as a second use of a moved-from object: the + // reassignment that makes it safe happens in an earlier iteration than the next move, and no + // analyzer can prove the correlation between the flag and the writers. (A swap is no better: + // it reads the slot too, so the checker relocates the finding to the move at the path append + // below.) The copy runs once per record boundary — and once per a merge's prefix empty — + // against a walk that spawns a git child per run, so it costs nothing measurable, and it + // leaves the slot's lifetime readable to man and machine alike. + if( haveLookahead ) { field = lookahead; haveLookahead = false; } else if( !fields.next( field ) ) { atEnd = true; break; } if( field.empty() ) { // One field of lookahead decides what this empty IS: the record boundary (the next field is // the marker, or the stream ends here) or a merge's prefix empty (anything else follows). + // The held-lookahead slot is ALWAYS empty here: the top of the loop consumes it (and clears the + // flag) before any field is examined, and the only writers — the two branches below — each + // assign a FRESH value before the next iteration. So the field after this empty is always the + // next one on the stream; there is no held field to prefer. A `haveLookahead ? … : …` here would + // be dead code, and its move of `lookahead` is also what made the slot's lifetime unreadable to + // bugprone-use-after-move (a second move in a later iteration that no reader can prove safe). std::string next; - bool haveNext = haveLookahead ? ( next = std::move( lookahead ), haveLookahead = false, true ) : fields.next( next ); + bool haveNext = fields.next( next ); if( !haveNext || next == kGitLogMarker ) { closeRecord(); From 0ced897610bf4c909ff99fc6506407a1d76e260d Mon Sep 17 00:00:00 2001 From: pt-act Date: Fri, 23 Oct 2026 23:59:59 +0100 Subject: [PATCH 08/15] fix(gitmine): the walkers' abandon contract rides the field read, per byte MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The migration to the NUL-framed reader narrowed the walkers' stop contract. The old line readers checked the abandon flag before every LINE; the field reader checked it only between FIELDS. For a stream that actually delivers NUL-terminated fields the two are equivalent, but a stream whose fields never arrive — a git answering in its default format, or a test shim streaming bare lines — has ONE unterminated field: the between-fields check never runs, and the read parks inside that field until end-of-stream instead of stopping at the next unit. Measured on the memory-guard fixture (memguardcheck B21): the pre-migration binary refused in 2s; the migrated reader blocked for the full 6s stream (the arm's limit is 4s). The flag is now polled once per byte inside readNulSafeField itself, which is where the old line reader's check lived. When it flips, the partial field is dropped and false is returned — which the caller already reads as end-of-stream, so it closes the pipe and that ends the child. false is ALSO end-of-stream, so a caller that must tell the two apart consults the flag itself, as the walkers already do at the top of their loop. After the fix the same arm refuses in about a second, faster than the line reader it replaces, because the poll is per byte rather than per line. The reader's contract is unchanged on healthy streams: the parameter defaults to null, the existing callers pass nothing, and the flag's load is relaxed — it is a cancellation signal, not a publication. --- src/gitmine.h | 18 +++++++++++++----- src/infra/stdinline.h | 12 +++++++++++- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/src/gitmine.h b/src/gitmine.h index 8ce573f19..2d6968d03 100644 --- a/src/gitmine.h +++ b/src/gitmine.h @@ -786,16 +786,20 @@ struct GitLogWalk // One field of lookahead over a NUL-framed stream: `next` is readNulSafeField or the held field, which // the boundary rule in forEachGitLogRecord parks here while it decides what an empty field means. +// `abandon` rides into the field read itself — the walkers' stop-at-the-next-field contract, polled +// once per byte, so a stream whose fields never arrive (a git answering in its default line format, or +// a shim streaming bare lines) cannot park the reader inside one field until end-of-stream. struct NulFieldStream { - std::FILE* pipe = nullptr; - std::string held; - bool hasHeld = false; + std::FILE* pipe = nullptr; + std::string held; + bool hasHeld = false; + const std::atomic* abandon = nullptr; bool next( std::string& field ) { if( hasHeld ) { field = std::move( held ); hasHeld = false; return true; } - return readNulSafeField( pipe, field ); + return readNulSafeField( pipe, field, abandon ); } }; @@ -853,7 +857,11 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin return walk; } - NulFieldStream fields{ pipe, {}, false }; + // The walkers' abandon contract rides into the field read: the flag is polled per byte, so a + // stream whose fields never arrive (the default-format degrade, a test shim's bare lines) stops + // the read where the old line reader stopped it — at its next unit — instead of blocking to + // end-of-stream inside one unterminated field. + NulFieldStream fields{ pipe, {}, false, &gGitWalksAbandoned }; std::string field; std::string lookahead; bool haveLookahead = false; diff --git a/src/infra/stdinline.h b/src/infra/stdinline.h index e6ba49f87..1c187abdb 100644 --- a/src/infra/stdinline.h +++ b/src/infra/stdinline.h @@ -32,6 +32,7 @@ #include "Diagnostics.h" // ASSUME — the null-stream precondition +#include #include #include @@ -145,7 +146,15 @@ inline bool readByteSafeLineBounded( std::FILE* in, std::string& line, std::size // parameter on the line reader because the two delimiters answer different questions ("where does this // line end" vs "is this field the record boundary") and a caller that confused them would read a // hostile filename as commit metadata. -inline bool readNulSafeField( std::FILE* in, std::string& field ) +// +// `abandon`, when non-null, is polled once per byte: the git history walkers stop at their next FIELD, +// and a stream whose fields never arrive — a git answering in its default line format, or a test shim +// streaming bare lines — must still abandon INSIDE the field read rather than park the reader in one +// never-terminated field until end-of-stream. When it flips, the partial field is dropped and false is +// returned, which the caller reads as end-of-stream; it then closes the pipe, and that is what ends +// the child. false is ALSO end-of-stream, so a caller that must tell the two apart consults the flag +// itself — which is what the walkers already do at the top of their loop. +inline bool readNulSafeField( std::FILE* in, std::string& field, const std::atomic* abandon = nullptr ) { ASSUME( in != nullptr ); @@ -153,6 +162,7 @@ inline bool readNulSafeField( std::FILE* in, std::string& field ) bool didReadAnyByte = false; for( int byteOrEof = std::fgetc( in ); byteOrEof != EOF; byteOrEof = std::fgetc( in ) ) { + if( abandon != nullptr && abandon->load( std::memory_order_relaxed ) ) { return false; } didReadAnyByte = true; if( byteOrEof == '\0' ) { From 8cab2253f4565af4e3fac07d2f06bbf49fa4ff90 Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 00:00:50 +0100 Subject: [PATCH 09/15] =?UTF-8?q?test(gitframingcheck):=20the=20one-line?= =?UTF-8?q?=20verdict=20wraps=20=E2=80=94=20gateexitcheck=20G2=20reads=20a?= =?UTF-8?q?=20single=20physical=20line?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gateexitcheck's G2 joins backslash-continuation lines into logical lines and flags a logical line containing `&& ok … || no …` only when it is a SINGLE physical line — the idiom is fine wrapped, because then the reader sees the verdict's shape, and a one-liner is what reads as an accident. Arm 5's files= verdict was a one-liner; it wraps now, matching the verdicts around it. Test-only. --- test/gitframingcheck.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/gitframingcheck.sh b/test/gitframingcheck.sh index 8be90fa1d..67806d598 100755 --- a/test/gitframingcheck.sh +++ b/test/gitframingcheck.sh @@ -418,7 +418,9 @@ marker_tracked=$( git -C "$H" ls-files | grep -cx '__C__' ) [ "$marker_tracked" = 1 ] && ok "a file named exactly __C__ is tracked — the name the frame must survive" \ || no "the marker-named file is not tracked ($marker_tracked)" files_n=$( grep -oE 'files="[0-9]+"' "$TMP/owners.xml" | head -1 | grep -oE '[0-9]+' ) -[ "$files_n" = 11 ] && ok "--owners analysed 11 files (files=11)" || no "owners files=$files_n, want 11" +[ "$files_n" = 11 ] \ + && ok "--owners analysed 11 files (files=11)" \ + || no "owners files=$files_n, want 11" ranked=$( grep -oE 'ranked="[0-9]+"' "$TMP/hotspots.xml" | head -1 | grep -oE '[0-9]+' ) unranked=$( grep -oE 'unranked_no_complexity="[0-9]+"' "$TMP/hotspots.xml" | head -1 | grep -oE '[0-9]+' ) [ "$ranked" = 11 ] && [ "$unranked" = 0 ] \ From 5c81ca551b652699768cdf3764537d5a5f2d5af7 Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 00:01:05 +0100 Subject: [PATCH 10/15] =?UTF-8?q?test(gitframingcheck):=20pin=20TZ=3DUTC?= =?UTF-8?q?=20=E2=80=94=20the=20CI=20digest/golden=20drift=20was=20the=20h?= =?UTF-8?q?ost's=20timezone,=20not=20git?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #433's gitframingcheck failure was read at first as git-version drift (CI's 2.55/2.56 against the 2.52 the pin was recorded on). It is not: building git 2.56.0 from source and running the fixture under it reproduces the pinned digest EXACTLY, byte for byte. The variable is the machine's timezone. The fixture's dates are naive strings ("2026-06-01T12:00:00"), and git reads a naive GIT_AUTHOR_DATE in the local zone, so every %at/%ct moves with the host: the same fixture reads a823b914… under WEST and 9a112638… under UTC, on both 2.52 and 2.56. CI's runners are UTC; the recording machine was not. So the fixture was never hermetic, and both of its machine-facing pins moved together: the raw-stream digest (arm 2) and the honest-tree golden (arm d), whose at= is the fixture's HEAD sha and therefore its commit epochs. The gate now exports TZ=UTC before it builds anything, the digest pin is the UTC reading, and the golden is re-recorded under UTC from the pre-change binary — with the recording procedure proven first by reproducing the committed golden byte-for-byte under the machine's own zone. The new golden differs from the old one in the at= values alone; every churn, author, share and ranking is identical, which is what a uniform shift must produce. Verified across the axes the failure lived on: ALL PASS under an ambient TZ=Asia/Tokyo, under TZ=UTC, under the machine default, and under both git 2.52.0 and the locally built 2.56.0. The pin lives HERE rather than in test/lib/clean-env.sh: that file is sourced by gates whose own fixtures (mergechurncheck and its 2026-06 dates) were derived on a non-UTC machine, and shifting them is a separate change with its own re-derivation. Those gates carry the same latent defect; the sweep for them is follow-up work, named here so it is not forgotten. --- test/gitframingcheck.sh | 10 +++++++++- test/gitframingcheck_golden.txt | 6 +++--- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/test/gitframingcheck.sh b/test/gitframingcheck.sh index 67806d598..47bd9ca1b 100755 --- a/test/gitframingcheck.sh +++ b/test/gitframingcheck.sh @@ -73,6 +73,14 @@ set -u ROOT="$( cd "$( dirname "$0" )/.." && pwd )" . "$ROOT/test/lib/clean-env.sh" +# (5) THE TIMEZONE. The fixture's dates are naive strings ("2026-06-01T12:00:00"), and git reads a +# naive GIT_AUTHOR_DATE in the MACHINE'S zone — so every %at/%ct in the stream moves with the host, +# and with them the digest pin below and the golden. Measured: the same fixture, the same command, +# the same git, reads a823b914… under WEST and 9a112638… under UTC — CI's runners are UTC, a dev Mac +# is not, and the gate failed on exactly that. Pin it HERE rather than in clean-env.sh: that file is +# sourced by gates whose own fixtures (mergechurncheck and its 2026-06 dates) were derived on a +# non-UTC machine, and shifting them is a separate change with its own re-derivation. +export TZ=UTC BIN="${1:-${RIPWIRE_BIN:-$ROOT/build/ripwire}}" # BOTH seams — positional and env [ "${BIN#/}" = "$BIN" ] && BIN="$ROOT/$BIN" GOLD="$ROOT/test/gitframingcheck_golden.txt" @@ -359,7 +367,7 @@ echo "arm 2: the raw stream the reader parses is byte-pinned" "$REAL_GIT" -c core.quotepath=false -C "$H" log --no-ext-diff --no-textconv -c \ -z --name-only --format='%x00__C__%x00%ae%x00%at%x00%ct' 2>/dev/null > "$TMP/raw.z" digest=$( python3 -c 'import hashlib,sys; print(hashlib.sha256(open(sys.argv[1],"rb").read()).hexdigest())' "$TMP/raw.z" ) -[ "$digest" = a823b914f6269ee4b50c23204c407c9e009940fa7d62d530d7316de418a33b3d ] \ +[ "$digest" = 9a11263860e97452932460b72c9eca9020a6ad01b1d29ceba917f1db1167796c ] \ && ok "the stream is byte-identical to the pinned digest" \ || no "raw-stream digest drifted: $digest" # Fields that spell the marker EXACTLY (NUL-delimited, so a '\n'-prefixed first path does not count): diff --git a/test/gitframingcheck_golden.txt b/test/gitframingcheck_golden.txt index d349a17da..b1920e1e4 100644 --- a/test/gitframingcheck_golden.txt +++ b/test/gitframingcheck_golden.txt @@ -1,6 +1,6 @@ ===== owners - + ===== hotspots - + ===== cochange - + From 6a6f4038621f550bd1723e02c523aff98560662b Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 00:01:21 +0100 Subject: [PATCH 11/15] test(crashsweep): the S2 registry names the one shared reader, not the five consumers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit crashsweepcheck's S2 static scan requires every popen/open/fdopen site in src/ to match an inline registry row (file, function, kind, count, reason) — stale rows fail too. The lane's five-consumers-to-one-reader refactor moved all five git-log walks onto forEachGitLogRecord's single popen, so five rows went stale at once and one new site went unregistered: six violations, all from the same refactor. The registry now names the shared reader — the walk loop leaves by stream end or the abandon break, and one pclose follows it on the only exit past the failed open. The gitCommandLines, popenTrimmed and os_win32 rows are untouched. Verified: crashsweepcheck ALL PASS, including all four S2 probes (the short-circuited fclose fires, an unregistered raw fopen fires, an owned-fd hand-off is not counted, an os::-spelled unregistered site still fires). --- test/crashsweepcheck.sh | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/test/crashsweepcheck.sh b/test/crashsweepcheck.sh index 206f7314d..419025b6a 100755 --- a/test/crashsweepcheck.sh +++ b/test/crashsweepcheck.sh @@ -303,13 +303,9 @@ darkflags.h readWhole fopen 1 closes fclose before the size-cap return and befor docparse.h openRegularFileStream fdopen 1 owned stored straight into pathguard::NoFollowRead, whose destructor fcloses it docparse.h openRegularFileStream open 1 owned fdopen'd into NoFollowRead; ::close only when fdopen declined it docparse.h runMarkitdown popen 1 closes no exit between popen and pclose +gitmine.h forEachGitLogRecord popen 1 closes the shared --name-only reader: the walk loop leaves by stream end or the abandon break; one pclose after it, on the only exit past the failed open gitmine.h gitCommandLines popen 1 closes status = pclose after the loop; no exit between -gitmine.h gitFileAuthors popen 1 closes continue-only loop; pclose after it -gitmine.h gitFileCommitCountsInDayWindow popen 1 closes continue-only loop; pclose after it -gitmine.h gitLogDecayedFileMining popen 1 closes continue-only loop; pclose after the flush infra/os_win32.cpp system popen 1 closes Windows body of os::system (compiled only for Windows): the drain loop has no exit; pclose after it -gitmine.h gitLogFileSets popen 1 closes continue-only loop; pclose after the flush -gitmine.h gitLogNameOnlyRaw popen 1 closes continue-only loop; pclose after it gitmine.h popenTrimmed popen 1 closes no exit between popen and pclose gitoracle.h loadOracleCache fopen 1 closes returns only on a failed open; fclose after the read loop gitoracle.h saveOracleCache fdopen 1 closes adopts ExclTempFile's released fd; fclose on its own line; a failed fdopen ::closes the fd From ef2ba396977c044e2d1bd2c3acb8bacf91a8f6a8 Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 00:01:22 +0100 Subject: [PATCH 12/15] test(deckcheck): --dashdash is a fixture's filename, not a flag MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit deckcheck requires every --token in the shipped prose to exist in --help or carry an allowlist row with a reason. The CHANGELOG's framing entry enumerates the hostile fixture's file names, one of which is `--dashdash.cpp` — a real filename in test/gitframingcheck.sh's fixture, quoted accurately, and not a ripwire flag. Rewording the prose to dodge the scanner would make the docs vaguer to satisfy a regex; the allowlist row is the honest fix, and it says what the token is. Unused allowlist rows fail this gate as rot, so the row carries the fixture reference that keeps it auditable. --- test/deckcheck_allowlist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/test/deckcheck_allowlist.txt b/test/deckcheck_allowlist.txt index eb6e22286..15b5aee91 100644 --- a/test/deckcheck_allowlist.txt +++ b/test/deckcheck_allowlist.txt @@ -140,3 +140,4 @@ --no-ext-diff git's own flag, named in the CHANGELOG's git-subprocess hardening entry as an option ripwire passes to its own git diff/log/show/blame children, not a ripwire flag --no-textconv git's own flag, same CHANGELOG hardening entry as --no-ext-diff above, not a ripwire flag --ignore-submodules git's own flag ("--ignore-submodules=dirty" on status/diff), same CHANGELOG hardening entry as --no-ext-diff above, not a ripwire flag +--dashdash not a flag at all — the leading bytes of the hostile fixture's FILENAME `--dashdash.cpp` (test/gitframingcheck.sh), quoted in the CHANGELOG's git-log framing entry where that fixture's file names are enumerated; a filename, never a ripwire flag From b3a64534ad0f54544512eaac15ca95bacea1b7ed Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 00:01:22 +0100 Subject: [PATCH 13/15] =?UTF-8?q?docs:=20CHANGELOG=20[Unreleased]=20?= =?UTF-8?q?=E2=80=94=20the=20framing=20fix's=20walkers=20stop=20at=20their?= =?UTF-8?q?=20next=20unit=20again?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The framing entry gains the follow-up its first cut earned: the NUL reader polls the walkers' abandon flag per byte, where the line readers it replaced polled per line, so a stream whose fields never arrive no longer parks the read inside one unterminated field until end-of-stream. --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index dc1f4353a..8e6168e35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -171,6 +171,13 @@ reader's stream, which for the repositories this fix targets is the corrupted on tell that blob from a fresh read. Every repository's history is walked once more after the upgrade; on trees with ordinary names the walk produces the same stream, so the cost is one cold walk and no changed answer. +**The walkers' stop contract rides the field read.** The line readers this replaces checked the abandon flag +before every line; the first cut of the field reader checked it only between fields, so a stream whose fields +never arrive — a git answering in its default format, or a shim streaming bare lines — held the read inside one +unterminated field until end-of-stream instead of stopping at the next unit. The flag is now polled once per +byte, and the memory-guard fixture that measures it refuses in about a second where the narrowed read blocked +for the whole stream. + **One case is disclosed, not fixed:** a merge whose FIRST combined path is a file literally named `__C__` still reads as pathless — its prefix empty is followed by the marker-spelled path, which the boundary rule cannot tell from a real record boundary, so that merge's files lose their attribution (a pathless phantom From 7d8b0a54579542d92d21ba3fe4a3b94324f2e046 Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 01:59:33 +0100 Subject: [PATCH 14/15] =?UTF-8?q?fix(gitmine):=20an=20empty=20metadata=20f?= =?UTF-8?q?ield=20is=20data=20=E2=80=94=20and=20ownership=20stops=20discar?= =?UTF-8?q?ding=20empty-email=20authors?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #433 review (Major), executed by the PM seat under the operator's 2026-10-24 sanction (producer in rate-limit cooldown; disclosed in the validation record). The boundary rule ate a legitimately empty %ae (git accepts --author="No Email <>") as a merge prefix, shifting the whole record — email<-the %at digits, epochs off by one, the first path's '\n' unstripped — into a pathless phantom; gitFileAuthors' carried-over line-frame guard then skipped any record with an empty email at all, so the file lost its ownership entirely (measured: "git unavailable / no history" on a healthy one-commit repo). The lookahead now decides what an empty is: followed by the MARKER (or end of stream) it is a record boundary whatever is pending — which is also what closes the collision phantom without swallowing the record after it (CodeRabbit's literal guard broke exactly there; the cascade was caught by arm (a) before it shipped). Followed by anything else while the marker or metadata is still owed, the empty is delivered to the state machine as the field it is; only a record owing no metadata reads a non-boundary empty as a merge prefix. The ownership guard keeps its epoch half alone (git never stamps a real commit at or before epoch 0) — the email half discarded real history. parseEpochField reports a degraded read (empty, non-digits, or leftover bytes) as nullopt and the walk counts it in GitLogWalk::epochUnparsed — the sink a consumer reads; a window that cut on a degraded epoch counted a floor, and saying so is the consumer's next step. Gate arm (f): a one-commit empty-email fixture, proven RED on the pre-fix binary alongside the full 40-arm battery, golden and digest unchanged. --- src/gitmine.h | 59 ++++++++++++++++++++++++++++++++--------- test/gitframingcheck.sh | 34 ++++++++++++++++++++++-- 2 files changed, 78 insertions(+), 15 deletions(-) diff --git a/src/gitmine.h b/src/gitmine.h index 2d6968d03..0fc70f5f7 100644 --- a/src/gitmine.h +++ b/src/gitmine.h @@ -782,6 +782,11 @@ struct GitLogWalk bool unread = false; bool notRun = false; int exitCode = 0; + // The degraded-epoch count — records whose author or committer epoch field was empty or + // unparsable (parseEpochField nullopt): their epochs read 0, so a window cut on them excluded + // those commits and its count is a FLOOR. This field is the sink a consumer reads to disclose + // that; folding it away would be the silent-floor the honesty rules forbid. + int epochUnparsed = 0; }; // One field of lookahead over a NUL-framed stream: `next` is readNulSafeField or the held field, which @@ -806,13 +811,17 @@ struct NulFieldStream // The marker as a whole field. A path may carry these bytes; only a field that IS this string opens a record. inline constexpr const char* kGitLogMarker = "__C__"; -// An epoch field: git's decimal seconds, 0 when the field is empty or does not parse. -inline std::int64_t parseEpochField( const std::string& field ) noexcept +// An epoch field: git's decimal seconds. nullopt when the field is empty, carries non-digits, or +// leaves bytes unconsumed — a DEGRADED read the walk counts (GitLogWalk::epochUnparsed) instead of +// folding it into a genuine zero, because epoch 0 is 1970 and can be real, while a failed parse +// means a commit whose window position is unknown: it falls below every cutoff, and any count that +// excluded it is a floor until the consumer says so. PR #433 review. +inline std::optional parseEpochField( const std::string& field ) noexcept { - if( field.empty() ) { return 0; } + if( field.empty() ) { return std::nullopt; } char* end = nullptr; const std::int64_t v = std::strtoll( field.c_str(), &end, 10 ); - return ( end != field.c_str() ) ? v : 0; + return ( end != nullptr && end != field.c_str() && *end == '\0' ) ? std::optional{ v } : std::nullopt; } // A pclose()/wait status as an exit code: WEXITSTATUS on a normal exit, -1 for a signal or an unreadable status. @@ -898,6 +907,16 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin if( haveLookahead ) { field = lookahead; haveLookahead = false; } else if( !fields.next( field ) ) { atEnd = true; break; } + // An empty field's meaning is decided by ONE field of lookahead, and the pending-metadata + // state changes only the non-boundary reading: an empty followed by the MARKER (or the end + // of the stream) is a record boundary no matter what is pending — that is also how a + // truncated phantom record (the disclosed collision) is closed without swallowing the record + // after it. An empty followed by anything else is DATA while the record still owes its + // marker or metadata — git accepts an empty author email (--author="No Email <>"), so %ae + // can arrive legitimately empty, and eating it as a merge prefix shifted the whole record + // (email←the %at digits, epochs off by one, the first path's '\n' unstripped) into a + // pathless phantom. Only a record that owes no metadata reads a non-boundary empty as a + // merge prefix. PR #433 review. if( field.empty() ) { // One field of lookahead decides what this empty IS: the record boundary (the next field is @@ -919,13 +938,20 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin empties = 0; havePath = false; if( haveNext ) { lookahead = std::move( next ); haveLookahead = true; } + continue; + } + if( wantMarker || metaLeft > 0 ) + { + // DATA: park the lookahead and fall through — the empty field itself is delivered to + // the state machine below as the pending metadata field it is (the empty-%ae shape). + lookahead = std::move( next ); haveLookahead = true; } else { ++empties; // a merge prefix (or the record's own close): skipped, but counted lookahead = std::move( next ); haveLookahead = true; + continue; } - continue; } if( wantMarker ) @@ -936,9 +962,14 @@ inline GitLogWalk forEachGitLogRecord( const std::string& root, const std::strin } if( metaLeft > 0 ) { - if( metaLeft == 3 ) { cur.authorEmail = field; } - else if( metaLeft == 2 ) { cur.authorTs = parseEpochField( field ); } - else { cur.committerTs = parseEpochField( field ); } + if( metaLeft == 3 ) { cur.authorEmail = field; } + else + { + const std::optional epoch = parseEpochField( field ); + if( !epoch ) { ++walk.epochUnparsed; } + if( metaLeft == 2 ) { cur.authorTs = epoch.value_or( 0 ); } + else { cur.committerTs = epoch.value_or( 0 ); } + } --metaLeft; continue; } @@ -2570,11 +2601,13 @@ inline std::vector gitFileAuthors( forEachGitLogRecord( root, extraArgs, [ & ]( const GitLogRecord& rec ) { // The old line frame parsed `email|ts` out of one field and skipped the commit's files when - // either half was missing or the epoch did not parse. Under the NUL frame the fields arrive - // separately, so the same guard is an empty email or a non-positive epoch — git never stamps a - // real commit at or before the epoch, and a header that forged one is exactly what this frame - // exists to make impossible. - if( rec.authorEmail.empty() || rec.authorTs <= 0 ) + // either half was missing or the epoch did not parse. Under the NUL frame the epoch half of + // that guard stands alone: git never stamps a real commit at or before the epoch, and a + // header that forged one is exactly what this frame exists to make impossible. The EMAIL half + // had to go — git accepts an empty author email (--author="No Email <>", PR #433 review), and + // the frame now delivers such a record intact, so skipping on an empty email would discard + // real history; the ownership key for it is honestly empty. + if( rec.authorTs <= 0 ) { return; } diff --git a/test/gitframingcheck.sh b/test/gitframingcheck.sh index 47bd9ca1b..16d37cb21 100755 --- a/test/gitframingcheck.sh +++ b/test/gitframingcheck.sh @@ -40,6 +40,10 @@ # (c2) the single-file pathspec seam — --owners=SYM appends a pathspec to the shared command, and a # pathspec glued to -z answers EMPTY, which reads as "git unavailable / no history" rather than # as a broken command. The pin for the one space that separates them. +# (f) an empty METADATA field is data, not framing — git accepts an empty author email +# (--author="No Email <>"), and the boundary rule must not eat a pending-metadata empty as a +# merge prefix: pre-fix the whole record shifted (email←the %at digits, epochs off by one, the +# first path's '\n' unstripped) and the commit was left PATHLESS, its file unattributed. # (d) the honest tree is byte-identical to the golden RECORDED FROM THE PRE-CHANGE BINARY — the # "no behavior change on honest trees" claim, over --owners/--hotspots/--cochange. # (e) the marker literal lives ONLY inside the shared helper — five consumer bodies, zero hits. @@ -94,8 +98,8 @@ command -v git >/dev/null 2>&1 || { echo "git required"; exit 2; } REAL_GIT="$( command -v git )" echo "gitframingcheck: BIN=$BIN ROOT=$ROOT git=$( git --version )" -TMP="$( mktemp -d )"; H="$( mktemp -d )"; G="$( mktemp -d )" -trap 'rm -rf "$TMP" "$H" "$G"' EXIT +TMP="$( mktemp -d )"; H="$( mktemp -d )"; G="$( mktemp -d )"; E="$( mktemp -d )" +trap 'rm -rf "$TMP" "$H" "$G" "$E"' EXIT # The fixtures are FRESH mktemp paths every run, and that is load-bearing beyond tidiness: the tool # memoizes its history walks under $TMPDIR/ripwire keyed on (repo, HEAD, window) — the repo key is # path-derived — so a fixed fixture path would let a previous run's cached churn answer this run's @@ -329,6 +333,32 @@ sym_p=$( tr '>' '\n' < "$TMP/owners-sym.xml" | grep -oE ' "$E/plain.cpp" +git -C "$E" add -A >/dev/null +git -C "$E" commit -qm "e1: empty-email author" --author="No Email <>" >/dev/null +UC +"$BIN" --owners=emptyMailFn --detail=1 --no-cache "$E" > "$TMP/owners-empty.xml" 2>"$TMP/owners-empty.err"; e_rc=$? +[ "$e_rc" = 0 ] && ok "--owners=emptyMailFn exits 0" \ + || no "--owners=emptyMailFn exits $e_rc: $( head -1 "$TMP/owners-empty.err" )" +e_p=$( tr '>' '\n' < "$TMP/owners-empty.xml" | grep -oE ' "$TMP/for-empty.xml" 2>/dev/null +e_churn=$( churn_of_fn "$TMP/for-empty.xml" "emptyMailFn" ) +[ "$e_churn" = "1" ] \ + && ok 'plain.cpp churn=1 — the empty-email commit counts for its own file' \ + || no "plain.cpp churn='$e_churn', want 1 (pre-fix: no row, no churn)" + # ── ARM (d): the honest tree is byte-identical to the pre-change golden ──────────────────────────── echo "arm d: the honest tree is byte-identical to the golden recorded from the PRE-CHANGE binary" { From cc6b312f7d23eb1111238fb8fecca9eb94c42486 Mon Sep 17 00:00:00 2001 From: pt-act Date: Sat, 24 Oct 2026 08:40:58 +0100 Subject: [PATCH 15/15] test(gitframingcheck): arm (f)'s committer rides an allowlisted synthetic domain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ripwirepubliccheck 5b red on CI (all five first shards): cm@x.invalid is email-shaped outside the synthetic allowlist (x.com/example.com/example.invalid/t.com/test.com). The fixture committer is now cm@x.com, matching every other synthetic address in this gate. Gate-only — the binary and every pinned digest/golden are untouched; gitframingcheck 40/0 and ripwirepubliccheck exit 0 re-verified locally on the fix. --- test/gitframingcheck.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/gitframingcheck.sh b/test/gitframingcheck.sh index 16d37cb21..b92a403e3 100755 --- a/test/gitframingcheck.sh +++ b/test/gitframingcheck.sh @@ -341,7 +341,7 @@ sym_p=$( tr '>' '\n' < "$TMP/owners-sym.xml" | grep -oE ' "$E/plain.cpp" git -C "$E" add -A >/dev/null git -C "$E" commit -qm "e1: empty-email author" --author="No Email <>" >/dev/null