Repository navigation
Real-world validation campaign: 64 audit defects and 5 campaign findings fixed - #32
Merged
Merged
Conversation
Move REPOS and clonePinned into tests/real-repos.ts so the build ratchet and the upcoming command matrix read one catalog. Add one pinned repo per language family the engine claims and had no real-world proof for: Ruby (sinatra), C# (MediatR), C (jq), C++ (fmt), Vue (pinia), Svelte (realworld), PHP (Slim), Kotlin (okhttp), Go nested modules (opentelemetry-go), Elixir (plug), Lua (lazy.nvim) and Scala (os-lib). Each ratio is the one measured at the pin. Clones now keep 30 commits of history so delta, churn and coupling have something to compare against.
Inputs a real repository can hold that cost seconds to minutes or come out wrong: 100k blank lines ahead of an import (Rust, Java, C#, Ruby, C) or a markdown link reference, one line of 120k calls, an 8,000-character word in the embedding tokenizer, a name repeated 40k times in its own file, a 24 MB LSP response in 64 KB chunks, a delta over 2,000 changed leaves of a 100k-edge graph, a 2,000-deep JS literal indexed with and without workers, a file named __proto__, paths with spaces, unicode and a newline, CRLF with a BOM, gitignore negated and POSIX classes, and a diff line reading '++ b/x'. Generated into temp dirs, so it runs in CI. Most cases are red here and turn green with the fixes that follow.
tests/e2e-commands.test.ts builds each pinned repository's index once, then runs every CLI command and all 40 MCP tools against it. Each call must exit as expected, print valid JSON where it promises JSON, leave no stack trace, print the same bytes twice, stay in the repo's time budget and, over MCP, stay under the response cap on the wire. tests/oracles/real-world.ts compares the answers with independent tools: the scan with git ls-files and git check-ignore, grep with rg -F, refs with rg -lw, delta with git diff --name-only, churn with git log --numstat, scip with a protobuf reader, plus callers within refs, impact against the direct importers, the outline/symbol_at/find_symbol round trip and a self-replacing symbolic edit. Opt-in: pnpm test:e2e:commands. docs/real-world-validation-2026-10.md records what the first run found and how each finding is classified.
…alls A repository's own .codeindex/lsp.json names a command to spawn, with its arguments and environment. A cloned untrusted repo could commit one running 'sh -c …' and wait for refs --lsp, callers --lsp, lsp status --probe or an agent's find_references call. That file is now read only with --trust-lsp-config (lsp, callers, refs and codeindex mcp) or CODEINDEX_TRUST_LSP_CONFIG=1. Otherwise only the configs the user placed themselves count: CODEINDEX_LSP_CONFIG, then ~/.config/codeindex/lsp.json ($XDG_CONFIG_HOME aware). lsp status reports the repository file as ignoredConfig and an lsp: true query says why it stayed static. Behaviour change: a team that relied on a committed .codeindex/lsp.json must now pass --trust-lsp-config or set CODEINDEX_TRUST_LSP_CONFIG=1. lsp.json budgets (timeoutMs, startupTimeoutMs) must be positive numbers; zero used to time every request out before it was sent. Every git call runs with LC_ALL=C: under a French locale 'fatal :' never matched the stripped 'fatal: ' prefix and error strings varied by machine. A user ref starting with '-' (a delta base, a since window, a changedSince ref) is refused before git reads it as an option, and changedSince passes --end-of-options: '--output=/tmp/x' made git diff write a file.
Same answers, linear time, on inputs a real repository can hold: - import regexes (Rust mod/use, Java, Ruby require, C include, C# using) and markdown link reference definitions took their leading whitespace with ^\s* under /gm, which crossed newlines: every line start of a blank run rescanned the rest of it (22 s for 100k blank lines of Rust). Indentation now comes from the line itself. - regex-tier call collection re-ran the definition-introducer regex on the whole line prefix for every match (7.9 s on one line of 120k calls); it now walks back to the word before the call, with the same verdict. - dead code copied a name's line list on every occurrence in its own file. - wordpiece tried every end for every start (104 s for an 8,000-character word): pieces are bounded by the longest vocabulary entry, and a word over 100 characters is UNK, as in BERT. EMBED_VERSION 2 -> 3, so no cached vector computed the old way is reused. - the LSP framer copied its whole buffer on every chunk, and rescanned a dropped over-cap frame from its start on each chunk; chunks are merged once a frame is complete and the header search resumes where it stopped. - delta rebuilt the dependents map over every edge, twice, for each changed file; one index now serves every changed file.
- A .vue/.svelte/.astro record was checked against its extension's own grammar key, which is none: built with --no-ast (or before a grammar pull), its regex-tier record was reused at the AST tier until the file was edited. A .h checked only C while it may be parsed as C++. Every grammar an extension can be parsed with is now compared. - A file named __proto__ set the prototype of cache.json's and freshness.json's files object instead of adding a key, so the index never took its unchanged fastpath. - Git's racily clean rule: a cache entry whose mtime is not older than the cache.json that recorded it loses its stat key and is hashed. On a filesystem with 1-2 s timestamps (HFS+, FAT, network mounts) or a coarse kernel clock, a same-size edit in the tick the index was written in kept the stale record. Files written before the index are trusted as before, so a fresh clone costs nothing. Two tests that simulated a touch with a future mtime now use a past one: a future stamp is never trusted. - The self-index guard compared --out as a '/'-joined prefix, so it never matched on Windows nor for an --out spelled with '..' segments. - EXTRACTOR_VERSION 15 -> 16: records extracted before these fixes are rebuilt.
- The AST collector and findFirst recursed once per tree level: a 2,000-deep array literal overflowed the main thread's stack and fell back to the regex tier, while a worker (another stack size) extracted it at the AST tier, so the index depended on how many workers ran. Both walk an explicit stack in the same order, and the declaration walk stops at a fixed 256 nested containers, deciding the same way on every thread. A parse or walk that still fails is recorded on the file as astFallback instead of silently. - A worker that exited with code 0 before answering left the run waiting out the ten-minute deadlock guard; any exit before the queue drains now fails the parallel pass, which falls back to the sequential scan at once. - The engine module a worker imports is recognised by its name, engine.mjs: a consumer's my-engine.mjs bundle also ended with it. - walk pushed subdirectories in name order onto a LIFO stack and so walked them in reverse; a --max-files cap kept the end of the alphabet. They come off in name order now. scope.test.ts pinned the old order and is updated.
…repo - In a bracket expression, '^' negates like '!' (git's wildmatch): x[^a] ignored xa and kept xb, git's exact opposite. POSIX classes ([:digit:], [:alpha:], …) are understood instead of ending the class at their first ']', escapes and a leading ']' are literal, and an unknown class matches nothing, as in git. A differential test checks every verdict against git check-ignore (case-sensitive, as the engine always is). - A walk rooted below a repository's top level (--repo pkg) now applies what git applies there: .git/info/exclude and every .gitignore from the top level down, matched against the path from the top. A root '*.generated.ts' or 'gen/' rule used to leave that package's generated files indexed. scan --why names such a rule by its file relative to --repo (../.gitignore). The adversarial expectation for 'x^' was wrong: git's x[^a] ignores it.
…kdown edges - Go: the in-repo module with the longest path prefix owns an import, as the go toolchain decides; the importer's own module used to win outright, so github.com/org/repo/api/v2/types dangled from the root module while the path go rejects (into a nested module's directory) resolved. - tsconfig: in one directory tsconfig.json comes before the root tsconfig.base.json it extends, whose aliases shadowed the project's overrides. The extends cycle guard covers the chain being read only, so a diamond reads the shared base on both sides and the later one wins, as in tsc. A path alias substitutes '*' through a function: '$&' or '$$' in a specifier are path text, not replacement patterns. - Python: a relative import climbing past the repository root dangles (escapes-repo-root) instead of resolving to a root file; 'from . import x' in a directory with no __init__.py is a PEP 420 namespace package, not a broken import. - Markdown: a percent-encoded target (my%20file.md) is probed decoded too, and an inline link's destination keeps balanced parentheses (docs/a(1).md).
- impact, call paths, the tests map and surprises now count extends and implements as dependencies. Two Java classes of one package, one extending the other, were linked by extends alone, and impact of the base found no dependent: delta reported no blast radius and a test subclassing its subject did not cover it. Relation edges no longer suppress the pair's use edge either. - A use edge (a file naming another file's unique declaration) is drawn within one language family, and never for a name the file binds by a named import: a TS file importing DataLoaderConfig from npm was linked to a Python class of that name, and impact of the Python file listed it. - diffHunks counts each hunk's lines from its @@ header: an added line reading '++ b/x' appeared as a '+++ b/x' file header and moved every later hunk to file x. tests/quality/baseline.json and the pinned compat outputs are unchanged.
… verdicts - Dead code reads a declaring file's code only: a Go doc comment above its function named it and downgraded every documented function to uncalled. The exported API of a Go library package (under a go.mod, not main, not internal/) and the public types of a published Gradle or Maven library are roots, as a package.json entry point already was. - Complexity counts the branch words of the file's language: a TypeScript 'const match = …' or a 'when' variable scored as branches. Python's soft 'case' counts where it opens a match clause. risk ranks only files the window changed, like hotspots. - PageRank runs up to 200 iterations: 100 stopped before the convergence threshold on slowly mixing graphs. The 'betweenness skipped' note is returned with the artifacts and printed by index. - The repo map skips a file that does not fit the budget instead of ending there: one oversized top file left it empty. - refs finds a doc mentioning a name that begins or ends with '$'. - Search: a qualified query (README.qqqzzzmissing) exists only if every part does; its first part alone used to make it a match. Each bridge records whether a stem or a trigram found it, where a trigram scoring exactly 0.9 was labelled stem. Tokens keep letters and digits of every script: конфигурация and 日本語 were dropped, Œuvre lost its Œ. The embedding tokenizer keeps the vocabulary's own normalization.
…onboard's brief - search --semantic joined --index under the repo, so an absolute index's embeddings.bin and endpoint cache were never found and the cache went to a phantom <repo>/<absolute path>; it goes through indexDirPath like every other read. - explainSemantic compared the query with an index of another model or vector width as if they matched (intDot read the shorter length): it now ranks lexically and says why in the explanation's note. - onboard returned nothing when its memory could not be written (a read-only checkout, a symlinked .codeindex); the brief comes back with memoryError. - The MCP lsp_status output schema knows the 'user' config source and ignoredConfig (from the LSP trust change), and onboard's knows memoryError.
…e cases
Behaviour change: the CLI used to accept and ignore what a command does not
read. It now exits 2, saying what the command takes and the closest flag:
- a flag the command does not read (grep --limit returned 7 hits: grep's cap
is --max-hits; search --since x ran unfiltered);
- an unknown single-dash token (grep -n searched for '-n', grep -i foo said
'unknown flag: foo'); a pattern starting with '-' goes after '--';
- an argument too many (scan extra, graph x);
- an unknown command, now before the walk and the grammar warm-up.
The walk's flags (--repo, --out, --include, --scope, --index, …) are accepted
by every command. '<command> --help' and '-h' print that command's help and
the flags it reads.
Also:
- --kinds is a value flag for the leading-flag hoisting ('--kinds all
deadcode' read 'all' as the command); a test checks every value flag is
known to the hoisting.
- Number flags take whole numbers (--depth 0.5, --max-files 2.5 and an empty
--workers were accepted).
- --out - writes to stdout (it created a file named '-'); index and embed
build refuse it. --out, scip and embed pull's model.json are replaced
atomically. codeindex mcp --repo refuses a file.
- A reader closing the pipe early (| head) ends the command with exit 0
instead of an unhandled EPIPE stack trace.
- rewrite takes the command line as ONE argument (several words were joined
into a different command) and never folds codeindex's own leading flags
into it. In a BRE/ERE, '*' and '^' after an escaped LITERAL paren or bar
are no longer read as the start of a group. grep -R and a searched tree
holding a git submodule are refused, and a plain rg rewrite excludes hidden
paths, as rg does. The rg cases of rewrite.test.ts gain those exclusions,
and query-surfaces.test.ts the new message for a stray argument.
… paths
- --max-response-bytes measured the text alone, while a tool with an output
schema sends it JSON-escaped plus the same data as structuredContent: about
1.8x on the wire (a 990 KB graph left as 1.8 MB; spring-security's impact
as 1.6 MB under a 1 MB cap). The serialized result is measured now. The
refresh command a withheld response suggests quotes the repository path.
- Count arguments are integers of at least 1 (inputSchema type integer,
checked by validateArgs): dead_code and duplicated_literals took limit 0,
and limit 2.5 returned 2 items while saying shown: 2.5.
- check_rules returns the CLI's {errors, warnings, violations} object, with
an output schema, instead of the bare array. The CLI's --config falls back
to a path under --repo, where MCP's configPath looks. A rule's empty kind
list, which meant every kind, is refused.
- LSP answers under the repository's real path are the repository's: a repo
under /tmp (really /private/tmp on macOS) or reached through a symlink had
every reference dropped. protocol.ts stays filesystem-free; the session is
told the root's other spelling.
mcp-output, mcp and rules tests follow the check_rules shape and the integer
type.
…ect_root - Mermaid labels carried the directory name as written, with only '"' swapped: a directory named '<img src=x onerror=…>' went out verbatim (HTML to a renderer with securityLevel loose), a newline split the node definition and a backtick run closed the clustered view's fence. Labels use Mermaid's entity codes and fold line breaks into spaces. - SCIP Metadata.project_root was 'file://' + the path: a root with a space or a '#' was not a valid URI, and a Windows root read 'file://C:/…'. It is pathToFileURL(root).
…mically - The tar reader wrote whatever bytes remained for an entry: a truncated archive from a custom CODEINDEX_GRAMMARS_URL (no sha256 to catch it) installed a cut wasm. A short entry now fails the pull, which writes nothing. - scripts/fetch-grammars.mjs deleted the committed scripts/grammars before copying, so one missing node_modules package left it half-empty. It builds a staging dir and swaps it in once complete; a failure removes the staging dir and leaves the committed one alone. - The browser fs shim removed a non-empty directory without recursive, and renameSync(x, x) deleted the file. Both now behave as node: EISDIR without recursive, a no-op rename onto itself.
- engine.ts said the bundle runs as 'node engine.mjs <cmd>' through a guard that does not exist; the CLI is scripts/cli.mjs. browser/entry.ts named a tsup browser target that build-browser.mjs explains it is not. - The help gains the flags it never listed (--depth, --direction, --kind, --budget-tokens, --base, --staged, --json, --fail-on), so '<command> --help' shows them. The README covers --base, --direction, --budget-tokens, --project-root, --ignore-case, the new e2e command matrix, the pinned corpus and the adversarial suite. - rewrite.test.ts's equivalence check drops grep's 'Binary file ./.git/… matches' line as it already dropped .git paths: the index's timestamps occasionally held the pattern.
Found by the real-world campaign on jq: jv_dtoa.c writes '#include "stdio.h"', '"errno.h"', '"float.h"'… With no local file of that name the compiler falls back to the system directories, as for the angle-bracket form, but the resolver reported each as a dangling missing-include. The C standard library, the common POSIX headers and sys/, arpa/, netinet/, net/ paths are external now. jq's ratchet tightens from 4.6% to 2.5% dangling (5 left: two generated .inc files, a generated version.h, and vendored headers reached through -I flags only the build system knows).
Found by the real-world campaign on sveltejs/realworld, where 41% of the edges dangled and no route reached src/lib: - $lib (and $lib/…) is SvelteKit's alias for the project's src/lib, set in a tsconfig that lives under the uncommitted .svelte-kit/. Under a svelte.config.* it now resolves there, unless a committed tsconfig paths entry says otherwise. - ./$types is the module SvelteKit generates per route: external, not a missing module. realworld's ratchet tightens from 42% to 3% dangling (one README image link, see S-104), and it gains a src/routes -> src/lib cross edge.
Found by the real-world campaign on opentelemetry-go: '[^2]: Trivial changes include …' has the shape of a link reference definition, so its first word became a link target and the graph held 16 dangling doc links to 'Trivial', 'Jonathan', '[Emit'… A definition whose label starts with '^' is a footnote and is skipped. opentelemetry-go drops to 0 dangling edges; its ratchet tightens to 0.1%.
…not broken Found by the real-world campaign on okhttp and sveltejs/realworld: every README or docs link to an image (../assets/images/events@2x.png, logo.png) was a dangling missing-target, because images are never indexed and the resolver only knew indexed files. A link to a file that exists on disk is now external, as a link to a directory already was: no node to link to, nothing broken. okhttp keeps one dangling link, to a file that really is missing; its ratchet tightens to 0.1%, realworld's to 1% (0 left).
Found by the campaign's baseline run: on macOS the temp dir under /var is /private/var, and node gives a module's own URL as a real path, so the grammars dir found next to the bundle is reported under /private/var while the test expected the /var spelling. The engine reports the true location; the expectation now compares realpathSync(pkg). The default suite passes on macOS again.
pinia still has the 24 dangling links it had at the pin (VitePress root-absolute links and generated files, documented limits). Its edge count fell from 830 to 793 when use edges stopped crossing languages and following imported names, so the same 24 now weigh 3.03%. The ratchet moves to 3.5%.
The validation report gains the final checks (default suite, command matrix on 19 repositories, e2e ratchet, reproducible build, quality report), the dangling ratios before and after on the repositories the campaign fixed, and the benchmark against 2.31.5.
… edge cases Found by the final audit of this branch: - The EPIPE handler exited 0, erasing the exit code 1 that rules and delta --fail-on set after writing: a failing CI gate passed when its output was piped into a reader that stopped early. It exits with the code already set. - --out went through the atomic rename even when the target was a symlink or a device: 'graph --out link.json' replaced the link with a file and '--out /dev/stdout' never reached stdout. A target that is not a regular file is written through. - A response withheld only for its wire size compared the artifact on disk with that wire size, and so called a fresh graph.json stale and told the agent to re-index. The artifact is compared with the text it would hold. - A percent-encoded '..' in a markdown link (%2e%2e/x.md) skipped the repo-escape check, and the disk probe then looked outside the repository. The decoded path is checked too.
github-actions Bot
pushed a commit
that referenced
this pull request
Oct 12, 2026
## [2.31.6](v2.31.5...v2.31.6) (2026-10-12) ### Bug Fixes * harden codeindex against real-world repositories ([#32](#32)) ([0a0b8f2](0a0b8f2))
Contributor
|
🎉 This PR is included in version 2.31.6 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A full review of the engine, then a real-world campaign: every CLI command and all 40 MCP tools run on 19 public repositories pinned by commit (13 language families), with the answers checked against
git,ripgrep,git check-ignoreand a protobuf reader. Every defect found, by the review or by the campaign, got a failing test first, then its fix.Behaviour changes worth reading
.codeindex/lsp.jsonnames commands to spawn, so it is read only with--trust-lsp-config(onlsp,callers,refs,codeindex mcp) orCODEINDEX_TRUST_LSP_CONFIG=1. OtherwiseCODEINDEX_LSP_CONFIGand~/.config/codeindex/lsp.jsoncount, andlsp statusreports the repository file asignoredConfig.-xtoken or a stray positional exits 2 with a suggestion (grep --limit→--max-hits).<command> --helpprints that command's help. Unknown commands are refused before any walk.check_rules(MCP) returns the CLI's{errors, warnings, violations}object, with an output schema.rewritetakes exactly one argument, refusesgrep -Rand trees holding a submodule, and a plainrgrewrite excludes hidden paths.EXTRACTOR_VERSION15 → 16 andEMBED_VERSION2 → 3: indexes and embeddings are rebuilt once.Commits are
fix:/perf:; none carries!. To ship the two hardenings as a major instead, markfix(security)andfix(cli)with!.Fixes, by area
-never reach git as options; git runs in the C locale.deltarebuilding the dependents map per changed file.__proto__file names, git's racily-clean rule, the self-index guard.[^…]and POSIX classes as git reads them;.gitignorefiles above a subdirectory--repo.%20and parentheses in markdown links, C system headers, SvelteKit$lib/$types, footnotes, links to images.extends/implementscount as dependencies;useedges stay in one language and skip imported names; diff hunks counted from their headers.$names, qualified-query verdicts, explicit bridge labels, Unicode tokens.project_rootas a real URI, truncated grammar archives, atomic grammar fetch, browserfsshim.Validation
pnpm typecheck,pnpm test(2,201 tests),pnpm check:build,pnpm test:e2e(98) andpnpm test:e2e:commands(new, 1,140 checks on 19 repos) pass. The default suite was red on macOS before (a French-locale git message, a/private/varpath).pnpm quality:reportis identical to the baseline. Bench against 2.31.5 (scripts/bench/optimization.mjs): the repository and synthetic corpora move within ±10% with mixed signs over two runs; on the 14-file fixture the harness showed warm/incremental slower, but a direct A/B of the same operations is within noise (details in the report).--outreplacing symlinks and devices, a fresh artifact reported stale, an encoded..in markdown links); each is fixed with a test in the last commit.tests/adversarial.test.ts(runs in CI),tests/e2e-commands.test.tswithtests/oracles/real-world.ts(opt-in).docs/real-world-validation-2026-10.md.