Real-world validation campaign: 64 audit defects and 5 campaign findings fixed

README.md

@@ -468,7 +468,7 @@ codeindex scip    --repo . --out index.scip   # SCIP index (--out - for stdout)
 codeindex callers --repo .                    # per-symbol caller index
 codeindex hierarchy       --repo .            # type hierarchy (both directions)
 codeindex implementations Runnable --repo .   # who implements it, transitively
-codeindex callgraph buildGraph --repo . --depth 2
+codeindex callgraph buildGraph --repo . --depth 2 --direction in   # callers of callers (out | in | both)
 codeindex find    Client/send --repo .        # declarations: signature, doc, parent, span
 codeindex refs    backoff --repo .            # defs, bound call sites, referencing files
 codeindex outline src/client.ts --repo .      # one file's symbols, in declaration order
@@ -480,6 +480,7 @@ codeindex workspaces --repo . --check         # monorepo packages; undeclared si
 codeindex resolution --repo .                 # per-language import resolution health
 codeindex mermaid src/app --repo .            # module diagram around a module, dir or file
 codeindex hotspots --repo . --since "6 months ago"   # where work concentrates
+codeindex repomap --repo . --budget-tokens 800      # the most central files and their API, in ~800 tokens
 ```
 
 `index` keeps a `cache.json` next to the artifacts, and every read command
@@ -547,6 +548,19 @@ decided it, its size against `--max-bytes`, the
 `--scope`/`--include`/`--exclude` that filtered it out, or the skipped
 directory above it.
 
+Every command checks its arguments. A flag it does not read (`grep --limit`,
+whose cap is `--max-hits`), an unknown single-dash token (`grep -n`: a pattern
+that starts with `-` goes after `--`) or an argument too many (`scan extra`)
+exits `2` with what the command takes and the closest flag it knows; an
+unknown command is refused before any walk. `codeindex <command> --help` (or
+`-h`) prints that command's help and the flags it reads. The flags of the
+walk (`--repo`, `--out`, `--include`, `--exclude`, `--scope`, `--index`, …) are
+accepted by every command, and may come before it (`codeindex --repo . grep
+foo`). Number flags take whole numbers. `--out -` writes to stdout, and every
+file a command writes (`--out`, `scip`, `embed pull`) is replaced atomically.
+Piping output into a reader that stops early (`| head`) ends the command
+quietly with exit `0`.
+
 ### Naming a symbol
 
 `callers`, `hierarchy`, `implementations`, `callgraph`, `callpath` and `refs`
@@ -846,6 +860,12 @@ Every occurrence and relationship names a symbol the index defines, and
 its relationship pass only knows the documents it has already visited, in Go
 map order, so a relationship to a symbol of another document is reported as
 missing at random.
+
+`Metadata.project_root` is the repository's `file://` URI, percent-encoded.
+`--project-root <uri>` overrides it — pin it (`--project-root
+file:///src/repo`) for an index whose bytes do not depend on where the
+checkout sits.
+
 ## What git history says
 
 Four commands read the commit history rather than the code: `churn` (commits
@@ -899,6 +919,7 @@ every point comes with the reason that fired it.
 
 ```sh
 codeindex delta --repo .                  # the branch vs its merge-base with the default branch
+codeindex delta --repo . --base v2.0.0     # vs the merge-base with another ref (a branch, a tag, HEAD~5)
 codeindex delta --repo . --staged --json  # the staged changeset, as JSON
 codeindex delta --repo . --fail-on HIGH   # CI gate: exit 1 when a module scores HIGH
 ```
@@ -986,6 +1007,8 @@ scan otherwise, and both give the same answer:
   (a directory or a single file) is ANDed with `--include`/`--exclude`. Globs
   are rooted at the repo: `*.ts` matches root-level files only, `**/*.ts`
   matches at any depth.
+- **Case.** Matching is case-sensitive; `--ignore-case` makes it insensitive.
+  `--files-with-matches` keeps one hit per file (`grep -l` with evidence).
 - **Bounded output.** Results stop at `--max-hits` (default 200). When the cap
   cuts the list, stderr says so and gives the number of matching files. A line
   longer than 300 characters comes back as a window around the match, and
@@ -1231,7 +1254,7 @@ that problem, so — same doctrine as the embedding tier — you can point one a
 the repository and get its answer *alongside* the static one:
 
 ```jsonc
-// <repo>/.codeindex/lsp.json — presence of this file IS the opt-in
+// ~/.config/codeindex/lsp.json — presence of this file IS the opt-in
 {
   "version": 1,
   "servers": [{
@@ -1257,6 +1280,17 @@ job or a slow machine can retune them without editing a shared file.
 `CODEINDEX_LSP_CONFIG` points at a config elsewhere; set to `off`, `0` or an
 empty string, it disables the tier even when the repository has one.
 
+**A repository's own config needs your consent.** A config names commands to
+spawn, so `<repo>/.codeindex/lsp.json` is code that repository would run on
+your machine. It is read only with `--trust-lsp-config` (on `lsp status`,
+`callers --lsp`, `refs --lsp` and `codeindex mcp`) or
+`CODEINDEX_TRUST_LSP_CONFIG=1`. Without either, only the configs you put there
+yourself count — `CODEINDEX_LSP_CONFIG`, then `~/.config/codeindex/lsp.json`
+(`$XDG_CONFIG_HOME/codeindex/lsp.json` when set) — and `lsp status` reports the
+repository's file as `ignoredConfig`, while an `lsp: true` query says why it
+stayed static. A `timeoutMs` or `startupTimeoutMs` that is not a positive
+number is rejected.
+
 The TypeScript example disables its separate syntax server because a CLI
 query opens a short-lived session. Otherwise an early reference request can be
 answered before the semantic project is ready and return only the declaration.
@@ -1550,7 +1584,7 @@ its indexed equivalent, for agent harnesses that intercept shell commands
 $ codeindex rewrite 'grep -rn TODO src'
 codeindex grep TODO --scope src --ignore-dir .codeindex
 $ codeindex rewrite "rg -tpy -w 'def main'"
-codeindex grep '\bdef main\b' --include '**/*.py' --include '**/*.pyi' --ignore-dir .codeindex
+codeindex grep '\bdef main\b' --include '**/*.py' --include '**/*.pyi' --exclude '**/.*' --exclude '**/.*/**' --ignore-dir .codeindex
 ```
 
 It prints the replacement and exits `0`. When it has no opinion, it exits `1`
@@ -1564,16 +1598,22 @@ understands recursive `grep`/`egrep`, `rg` and `git grep`:
   `--include`/`-g` base-name glob becomes `**/<glob>`, and `-t` becomes the
   globs of that ripgrep type. `--ignore-dir .codeindex` turns off the default
   vendor/build/out/tmp skips, which none of these tools make. Gitignored files,
-  lockfiles and binaries are still left out, on purpose.
+  lockfiles and binaries are still left out, on purpose. A plain `rg` skips
+  hidden files and directories, so its rewrite excludes them too (`--hidden` or
+  `-uu` keeps them).
 - **The flags.** `-l` becomes `--files-with-matches`, and a pattern that starts
   with `-` goes behind `--`.
 
 The parser is deliberately conservative. The rewrite is refused when the line
 contains shell syntax outside single quotes (pipe, redirect, substitution,
 chaining, braces, an unquoted glob in a path), an unrecognized or
-output-changing flag (`rg -r` is `--replace`), a non-recursive `grep`, a path
-outside the tree, more than one path, include/exclude rules whose order
-matters, or regex syntax that cannot be translated exactly. A refusal costs
+output-changing flag (`rg -r` is `--replace`, `grep -R` follows symlinks the
+engine never does), a non-recursive `grep`, a path outside the tree, more than
+one path, include/exclude rules whose order matters, regex syntax that cannot
+be translated exactly, or a searched tree holding a git submodule (grep and rg
+search it, the engine stops at it). The command line is ONE argument:
+`codeindex rewrite grep -r foo` exits `1`, and flags given to codeindex before
+`rewrite` are not part of it. A refusal costs
 nothing, while a wrong rewrite would silently change what the agent asked for.
 The test suite runs each supported form through the real tool and through its
 rewrite, and checks that both find the same lines.
@@ -1743,8 +1783,16 @@ pnpm typecheck
 pnpm build         # tsup → scripts/engine.mjs + scripts/engine.d.mts
 pnpm check:build   # proves the committed bundle is byte-reproducible
 pnpm test:e2e      # opt-in: pinned real-repo builds with ratchets
+pnpm test:e2e:commands   # opt-in: every command and MCP tool on the pinned repos, against git/ripgrep oracles
 ```
 
+The real-world corpus (19 public repositories across 13 language families,
+each pinned by commit) lives in `tests/real-repos.ts`; `CODEINDEX_E2E_REPOS=a/b`
+narrows a run to some of them. `tests/adversarial.test.ts` holds the inputs that
+once cost minutes or came out wrong (blank-line runs, 120k calls on a line,
+deep nesting, odd file names), and runs with the default suite. The last
+campaign's findings are in `docs/real-world-validation-2026-10.md`.
+
 The compat suite pins golden bytes for the `mini-repo` fixture — the proof
 that extraction stays lossless across releases.
 

docs/real-world-validation-2026-10.md

@@ -0,0 +1,152 @@
+# Real-world validation — 2026-10
+
+Every CLI command and every one of the 40 MCP tools was run against 19 public
+repositories pinned by commit, covering 13 language families. The answers were
+compared with what independent tools say they must contain: `git`, `ripgrep`,
+`git check-ignore` and a protobuf reader. This report lists what that campaign
+found and how each finding was classified.
+
+Baseline: `d15d835` (2.31.5). Host: macOS, Apple Silicon, Node 24.19.0,
+git 2.x, ripgrep 14.
+
+## Corpus
+
+The seven repositories of the existing build ratchet, plus twelve pinned on
+2026-10-12, each chosen as an independent proof for a language family the
+engine claims. None of them was used to design a resolver.
+
+| Repository | Language | What it exercises |
+|---|---|---|
+| socialgouv/code-du-travail-numerique | TypeScript | Yarn workspaces, Next.js app dir |
+| t3-oss/create-t3-turbo | TypeScript | Turborepo + pnpm, conditional `exports` |
+| nrwl/nx-examples | TypeScript | Nx, root `tsconfig.base.json` aliases |
+| pallets/flask | Python | src layout, package imports |
+| gin-gonic/gin | Go | module-rooted imports |
+| BurntSushi/ripgrep | Rust | cargo workspace |
+| spring-projects/spring-security | Java + Kotlin | large Gradle multi-project |
+| sinatra/sinatra | Ruby | bare `require` across lib roots |
+| jbogard/MediatR | C# | namespace `using` across projects |
+| jqlang/jq | C | quoted includes, vendored libraries |
+| fmtlib/fmt | C++ | `include/fmt` headers, `.h` read as C++ |
+| vuejs/pinia | Vue + TypeScript | pnpm monorepo, VitePress docs |
+| sveltejs/realworld | Svelte | SvelteKit `$lib` alias, generated `./$types` |
+| slimphp/Slim | PHP | composer PSR-4 |
+| square/okhttp | Kotlin | Gradle multi-project, mkdocs docs |
+| open-telemetry/opentelemetry-go | Go | nested modules |
+| elixir-plug/plug | Elixir | module aliases |
+| folke/lazy.nvim | Lua | `require` onto `lua/` |
+| com-lihaoyi/os-lib | Scala | Mill layout, `src-2`/`src-3` |
+
+The catalog, with each pin and the dangling ratio measured at it, is
+`tests/real-repos.ts`.
+
+## Method
+
+`tests/e2e-commands.test.ts` (opt-in: `pnpm test:e2e:commands`) builds each
+repository's index once, then runs every command of `codeindex --help` and every
+MCP tool through one `codeindex mcp` session. It runs the built CLI, the surface
+users get.
+
+Every call is held to five invariants: the expected exit code, valid JSON on
+stdout where JSON is promised, no stack trace on stderr, byte-identical output
+on a second run, and the repository's time budget. MCP answers must also stay
+under the server's `--max-response-bytes` on the wire.
+
+Per repository, the oracles in `tests/oracles/real-world.ts` check:
+
+| Command | Compared with |
+|---|---|
+| `scan` | `git ls-files -co --exclude-standard`: every file git would build is indexed or skipped with a reason; nothing `git check-ignore` ignores is indexed |
+| `grep` (literal) | the hits of `rg -F` on the indexed files |
+| `refs` | every referencing file contains the name as a word (`rg -lw`); recall is reported |
+| `callers` | its files are a subset of `refs` |
+| `impact` | it contains every direct importer in graph.json |
+| `delta --base HEAD~5` | `git diff --name-only` |
+| `churn` | an independent parse of `git log --numstat` over the same window |
+| `workspaces` | every manifest exists on disk |
+| `scip` | it decodes, and `project_root` is a valid URI that round-trips to the repo |
+| `mermaid` | no raw angle bracket or unbalanced quote in a label |
+| `outline` / `symbol_at` / `find_symbol` | the three agree on every declaration of the most-imported files |
+| symbolic edits | replacing a declaration with its own body leaves the file byte-identical |
+
+`tests/adversarial.test.ts` complements it with a synthetic corpus that runs in
+CI: inputs that cost seconds to minutes, or came out wrong, before this campaign.
+
+## Findings
+
+Each finding is one of three things. A **bug** gets a failing test reduced to a
+minimal fixture, then a fix. An **oracle false positive** is a defect of the
+check, corrected in the oracle. A **documented limit** is a known boundary of a
+static, build-system-free indexer.
+
+### Bugs
+
+| Finding | Seen on | Step |
+|---|---|---|
+| An MCP answer with `structuredContent` goes out at about 1.8× its text, past `--max-response-bytes` (`impact` on spring-security: 1.6 MB against a 1 MB cap) | spring-security | S-020 |
+| A quoted include of a C standard header with no local copy (`#include "stdio.h"`) is reported dangling; the compiler falls back to the system headers | jq | S-101 |
+| SvelteKit's `$lib` alias is never resolved, and the generated `./$types` module is reported dangling: 41% of realworld's edges dangle and no `$lib` import reaches `src/lib` | sveltejs/realworld | S-102 |
+| GFM footnote definitions (`[^2]: Trivial changes…`) are read as link reference definitions, giving dangling links to `Trivial`, `Jonathan`, `[Emit` | opentelemetry-go | S-103 |
+| A markdown link to a file that exists but is not indexed (an image) is reported as a missing target | okhttp, sveltejs/realworld | S-104 |
+| `tests/index-cache.test.ts` compares a realpath with a tmpdir path and fails on macOS, where `/var` is `/private/var` | baseline run | S-105 |
+
+### Oracle false positives
+
+| Finding | Correction |
+|---|---|
+| A directory symlink is one blob to git and one `directory-symlink` skip to the walk | the oracle matches the skip by its exact path |
+| An uninitialised submodule is a gitlink and an empty directory | gitlinks (mode 160000) leave the git file list |
+| `find_symbol` was capped at 500 matches while fmt declares more than 500 `TEST` macros | the round trip asks for every match |
+| `delta` lists a changed lockfile under `unindexed`, not `changes` | the oracle reads `changes`, `unindexed` and `deleted` |
+
+### Documented limits
+
+| Finding | Why it stays |
+|---|---|
+| `#include "gtest/gtest.h"` (fmt) and `decNumber/decNumber.h` (jq) dangle | they resolve through `-I` directories that live in CMake or Make files; the engine reads no build system |
+| VitePress root-absolute links (`/core-concepts/index.md` in pinia's docs) dangle | they are relative to the docs site root, which only the site configuration knows |
+| Generated files (`api/typedoc-sidebar.json`, `vendor/autoload.php`) dangle | they do not exist in the checkout |
+| gtest `TEST(suite, name)` blocks are declarations named `TEST` | the C++ grammar parses a macro invocation with a body as a function definition |
+
+## Audit defects
+
+The 64 defects found by reading the code before the campaign are fixed in
+steps S-010 to S-023, each with a test that failed first: LSP config trust and
+git hardening, the quadratic and cubic paths, cache reuse, thread-independent
+extraction, gitignore semantics, import resolution, graph and diff semantics,
+analyses and search, embeddings, the CLI argument contract, MCP limits and
+response size, rendering, and tooling.
+
+## After the fixes
+
+| Check | Result |
+|---|---|
+| `pnpm typecheck`, `pnpm test` | pass — 2,198 tests (the baseline had 2 failing on macOS: a French-locale git message and a `/private/var` path, S-010 and S-105) |
+| `pnpm test:e2e:commands` | 1,140 of 1,140 on the 19 repositories; the only notes left are report-only `refs` recall misses (a Jest snapshot and a `package.json` naming the symbol as text) |
+| `pnpm test:e2e` | 98 of 98 |
+| `pnpm check:build` | the committed bundles are byte-reproducible |
+| `pnpm quality:report` | identical to the baseline |
+| `tests/adversarial.test.ts` | every case green; before the fixes, 100k blank lines of Rust took 10 s, one line of 120k calls 20 s, a 30 MB LSP response read in 16 KB chunks 5 s |
+
+Dangling ratios at the pins, before and after:
+
+| Repository | Before | After | Why |
+|---|---|---|---|
+| sveltejs/realworld | 11/27 (40.7%) | 0/44 | `$lib`, `./$types`, a README image |
+| jqlang/jq | 12/259 (4.6%) | 5/252 (2.0%) | quoted libc headers; the rest are generated `.inc` files and `-I` vendor paths |
+| open-telemetry/opentelemetry-go | 16/6,844 | 0/6,835 | footnotes |
+| square/okhttp | 6/6,032 | 1/6,028 | images; the one left is a link to a file that is not there |
+| vuejs/pinia | 24/830 | 24/793 | unchanged links; 37 cross-language or imported-name `use` edges left |
+
+Every ratchet in `tests/real-repos.ts` was re-pinned to the measured value.
+
+Performance, measured with `scripts/bench/optimization.mjs` (2.31.5 against
+this branch, same source bytes, 5 runs per mode, run twice): on the repository
+itself and the synthetic corpus, cold, warm and incremental index, query and
+MCP times moved within ±10% with mixed signs between the two runs. On the
+14-file mini fixture the harness showed warm and incremental index 8–59%
+slower, but a direct A/B of the same operations (10 runs each, twice) gave
+72.9/75.7 ms against 77.2/73.6 ms warm and 114.0/100.8 ms against 113.1/101.4 ms
+incremental, within noise. A `refs` query on that fixture costs 1.6 µs instead
+of 0.9 µs (the `$`-aware mention boundary); on the larger corpus it is 16%
+faster.

package.json

@@ -40,6 +40,7 @@
     "test": "node scripts/fetch-grammars.mjs --extended && vitest run",
     "test:watch": "node scripts/fetch-grammars.mjs --extended && vitest",
     "test:e2e": "node scripts/fetch-grammars.mjs --extended && CODEINDEX_E2E=1 vitest run tests/e2e-real-repos.test.ts",
+    "test:e2e:commands": "node scripts/fetch-grammars.mjs --extended && CODEINDEX_E2E=1 vitest run tests/e2e-commands.test.ts",
     "quality": "node scripts/fetch-grammars.mjs --extended && CODEINDEX_QUALITY_WRITE=1 vitest run tests/quality.test.ts",
     "quality:report": "node scripts/fetch-grammars.mjs --extended && CODEINDEX_QUALITY_REPORT=1 vitest run tests/quality.test.ts",
     "typecheck": "tsc --noEmit",

scripts/cli.mjs

@@ -2,6 +2,12 @@
 
 // src/cli-entry.ts
 import { runCli } from "./engine.mjs";
+for (const stream of [process.stdout, process.stderr]) {
+  stream.on("error", (e) => {
+    if (e.code === "EPIPE") process.exit(process.exitCode ?? 0);
+    throw e;
+  });
+}
 try {
   await runCli(process.argv.slice(2));
 } catch (e) {