idasql-inspired analysis layer: SQLite index, SQL query, Ghidra daemon

.claude/CLAUDE.md

@@ -1,18 +1,10 @@
 # Vibe Reverse Engineering -- Claude Code Instructions
 
-## Read-Only Templates
+Shared conventions (project overview, read-only templates, workspace/backup/KB rules, engineering standards, code comments) are canonical in the root file, auto-loaded here:
 
-These directories are **shared tooling and templates**. Do not modify them for game-specific work — per-game changes go in `patches/<GameName>/`.
+@../AGENTS.md
 
-- `rtx_remix_tools/dx/remix-comp-proxy/` — proxy framework **template** (copied per-game)
-- `rtx_remix_tools/dx/scripts/` — DX9 analysis scripts (shared tooling)
-- `retools/` — static analysis toolkit (shared tooling)
-- `livetools/` — Frida-based dynamic analysis (shared tooling)
-- `graphics/` — DX9 tracer framework (shared tooling)
-
-**Per-game work goes in `patches/<GameName>/`.** When starting a new game, copy `rtx_remix_tools/dx/remix-comp-proxy/` (excluding `build/`) to `patches/<GameName>/` and edit the copy. If the user says "edit remix-comp-proxy code" without specifying, ask whether they mean the template or a game copy.
-
-Shared tooling can be modified to improve the tools themselves — just not for game-specific customization.
+The sections below are Claude Code-specific.
 
 ---
 
@@ -30,58 +22,6 @@ The main agent owns `livetools` — always use them to verify static findings, p
 
 ---
 
-## Engineering Standards
-
-Every change should make the codebase better, not just make the problem go away. If a solution needs a paragraph to justify why it's not a hack, it's a hack.
-
-### Remove
-- **Fixes in the wrong layer**: a guard on a canvas to suppress commits that a model should own. Put the fix where the problem originates.
-- **Tolerance inflation**: widening deltas or adding retries to hide flaky behavior. If the value is wrong, find out why.
-- **Catch-all exception swallowing**: `try/except Exception: pass` to hide symptoms.
-- **Excessive error/null handling**: adding too many error/None "if" checks. If the error is expected, handle it. If unexpected, raise it.
-- **God methods**: 200+ line functions doing multiple things. Break into named steps. Focus on cognitive load. Design for fewer indentation levels.
-- **Leaky abstractions**: implementation details leaking into layers/modules that should be agnostic of one another.
-
-### Design For
-- **Single responsibility**: one component, one job. If you need "and" to describe it, split it.
-- **Ownership**: the component that creates the problem owns the fix.
-- **Minimal public surface**: expose what consumers need, nothing more.
-
-### Commit to the New Code
-- **No legacy fallbacks**: if you replace a system, remove the old one.
-- **No dead code**: commented-out blocks, unused imports, orphan functions "just in case". Version control is the safety net.
-- **No multiple paths to the same result**: one way to do each thing. If two paths exist, one is wrong.
-- **No half-migrations**: finish the job -- update every reference, remove old APIs.
-
-### Smell Tests
-- "It works if I add a sleep" -- broken data flow.
-- "It works if I read from widget instead of storage" -- the two are out of sync.
-- "It passes alone but fails with other tests" -- shared mutable state leaking.
-- "I added a flag to skip this code path" -- why does that path run in the first place?
-
-## Code Comments
-
-Each file reads as if it was always designed this way. Comments guide the next developer, not narrate the development journey.
-
-### Remove
-- **Implementation backstories**: "We do this because the other day X happened"
-- **Obvious narration**: "Create the attribute", "Loop through keys", "Check if valid" -- if the code says it, the comment is noise
-- **Debugging breadcrumbs**: "Without this, subsequent tests may see the modifier key as still held"
-- **Trial-and-error reasoning**: "We tried X but it caused Y so we do Z instead"
-
-### Keep
-- **Non-obvious design decisions**: stated as *what* and *why this design*, not *what happened to us*
-- **Tricky invariants**: conditions that would be easy to accidentally break
-- **API contracts**: docstrings on public methods with Args, Returns, Raises
-
-### Prefer Instead
-- **Rename** a variable or function to be self-explanatory rather than adding a comment
-- **Docstrings** on classes and public methods (Google style: `Args:`, `Returns:`, `Raises:`)
-- **Type hints** over comments about expected types
-- **Short inline comments** on the *why*, never the *what*
-
----
-
 ## DX9 FFP Porting
 
 Invoke the **`dx9-ffp-port` skill** before editing `renderer.cpp`, `ffp_state.cpp`, `remix-comp-proxy.ini`, or draw routing; porting a game for RTX Remix; diagnosing VS constants, vertex declarations, matrix mapping, or skinning; or building/deploying a remix-comp-proxy patch.

.claude/agents/static-analyzer.md

@@ -27,38 +27,66 @@ test -f retools/data/signatures.db || python retools/sigdb.py pull
 ```bash
 grep -cE '^[@$]|^struct |^enum ' patches/<project>/kb.h 2>/dev/null || echo 0
 ```
-If the count is under 50 (or the file doesn't exist), run `python -m retools.bootstrap <binary> --project <Project>` first. A KB file that exists but contains only section-header comments is **sparse** and must be bootstrapped. Do not skip bootstrap just because the file exists.
+If the count is under 50 (or the file doesn't exist), run `python -m retools.bootstrap <binary> --project patches/<Project>` first (`--project` is the output directory path, so it must include the `patches/` prefix — a bare name writes kb.h outside the project tree). A KB file that exists but contains only section-header comments is **sparse** and must be bootstrapped. Do not skip bootstrap just because the file exists.
 
 **4. Ghidra project**: Check if a Ghidra project exists for the binary:
 ```bash
 python retools/pyghidra_backend.py status <binary> --project patches/<Project>
 ```
 If "Not analyzed", run `python retools/pyghidra_backend.py analyze <binary> --project patches/<Project>`. Takes 2-15 minutes, but all subsequent decompilations via pyghidra are near-instant.
 
+**5. Index**: Check whether the project has an index.db and what's in it before scanning the binary yourself:
+```bash
+python -m retools.index status <Project>
+```
+If `funcs`/`xrefs` show `source='bootstrap'` only (or the table is empty), and a Ghidra project exists, run `pyghidra_backend.py export` to seed authoritative facts — see "Query-first workflow" below.
+
 ## Running Tools
 
 Run all tools from the repo root. Use `python -m retools.<module>` or `python retools/<module>.py` syntax:
 
-### Decompilation (two backends)
+### Decompilation -- Ghidra primary, r2ghidra fallback
 
-**pyghidra (preferred when Ghidra project exists)** — better MSVC type propagation, library call resolution, larger function scope detection:
+**pyghidra is the primary backend** once a Ghidra project exists — better MSVC type propagation, library call resolution, larger function scope detection, and its facts can be exported into `index.db` for instant SQL lookups later:
 ```
 python retools/pyghidra_backend.py decompile binary.exe 0x401000 --project patches/proj
 ```
 
-**r2ghidra (fast fallback)** — better `__thiscall` on small functions, no JVM startup:
+**r2ghidra is the zero-setup fallback and second opinion** — no Ghidra install required, better `__thiscall` recovery on small functions, no JVM startup, and useful to cross-check a pyghidra decompile that looks wrong:
 ```
 python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h
 python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h --backend pdg
 ```
 
-**Auto mode (tries pyghidra first, falls back to r2ghidra)**:
+**Auto mode (tries pyghidra first, falls back to r2ghidra)** — routing unchanged:
 ```
 python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h --project patches/proj
 ```
 
 When told to use a specific backend, use it. Otherwise prefer auto mode with both `--types` and `--project`.
 
+**Ghidra daemon**: if `python -m retools.ghidra_server <Project>` is running (port 27043; livetools owns 27042), `decompile`/`export`/`kb-apply` route through it automatically and repeat calls become sub-second. Warm the server yourself before a batch of decompiles on the same project: `python -m retools.ghidra_server <Project> --idle 600` (background it). `RETOOLS_GHIDRA_COLD=1` or `--cold` forces a cold in-process run when you need to bypass the daemon. The daemon records its pid/port/project/binary in `patches/<Project>/ghidra/.state.json`, deleted on shutdown once the Ghidra program is closed and the Windows `.rep` lock is released.
+
+### Query-first workflow
+
+Before re-scanning a binary with xrefs/datarefs/search/funcinfo, check whether `index.db` already has the answer — a SQL query against a local file is cheaper than re-disassembling:
+
+```bash
+python -m retools.index status <Project>                          # per-table counts + schema_version
+python -m retools.query <Project> --list-tables                    # confirm what's queryable
+python -m retools.query <Project> --schema funcs                   # PRAGMA table_info before writing joins
+python -m retools.query <Project> "SELECT * FROM callers WHERE callee_addr=0x401000"
+python -m retools.query <Project> "SELECT * FROM grep WHERE name LIKE '%Ground%'" --json
+```
+
+Only fall back to `xrefs.py`/`datarefs.py`/`search.py`/`funcinfo.py` for facts `index.db` doesn't have yet (e.g. no `export` has run, or the question needs a live disassembly detail not captured by the schema).
+
+**Hard pushdown rule**: `decompile` and `export` require a specific function address (or, for `export`, an analyzed program) — never invoke them without one, or you decompile/scan the whole binary instead of the function you actually need. If you don't have an address yet, get one from `query`, `search`, or `xrefs` first.
+
+**Read-First mutation discipline**: `kb-apply` mutates the Ghidra project. Always decompile or `query` the target function first to confirm the current name/prototype, run `kb-apply`, then **re-decompile the same function** to verify the change landed before reporting it as done. `kb-apply` is idempotent — re-running it should produce stable counts and no errors, so if a second run changes anything, treat that as a bug, not expected behavior.
+
+**Cost guard**: run `export` once per analysis pass (after `kb-apply`, so exported names reflect it), not once per query — repeated `export` calls re-walk the whole program for no benefit once `index.db` is current.
+
 ### Other tools
 ```
 python -m retools.search binary.exe strings -f "error" --xrefs
@@ -67,13 +95,18 @@ python -m retools.callgraph binary.exe 0x401000 --up 3
 python -m retools.structrefs binary.exe --aggregate --fn 0x401000 --base esi
 python -m retools.dumpinfo crash.dmp diagnose --binary d3d9.dll
 python -m retools.throwmap d3d9.dll match --dump crash.dmp
-python -m retools.bootstrap binary.exe --project MyGame
+python -m retools.bootstrap binary.exe --project patches/MyGame
 python -m retools.sigdb scan binary.exe --db retools/data/signatures.db
 python -m retools.sigdb identify binary.exe 0x401000 --db retools/data/signatures.db
 python -m retools.sigdb fingerprint binary.exe
 python -m retools.context assemble binary.exe 0x401000 --project MyGame
 python retools/pyghidra_backend.py analyze binary.exe --project patches/MyGame
 python retools/pyghidra_backend.py status binary.exe --project patches/MyGame
+python retools/pyghidra_backend.py export binary.exe --project patches/MyGame
+python retools/pyghidra_backend.py kb-apply binary.exe --project patches/MyGame --kb patches/MyGame/kb.h
+python -m retools.index status MyGame
+python -m retools.query MyGame "SELECT * FROM funcs WHERE name LIKE '%Update%'"
+python -m retools.ghidra_server MyGame --idle 600
 ```
 
 If `retools/data/signatures.db` is missing, run `python -m retools.sigdb pull` to download it.
@@ -140,3 +173,16 @@ Also update `patches/<project>/kb.h` with any new function signatures, structs,
 In your return message, state the file path you wrote to and give a brief summary. The main agent will read the file for full details.
 
 Update your agent memory with significant architectural discoveries, identified subsystems, and class hierarchies that will be useful in future sessions.
+
+## Routing to Adjacent Skills/Docs
+
+This agent owns offline static analysis. Hand off to the right reference/skill instead of improvising:
+
+| Need | Go to |
+|------|-------|
+| Full tool syntax, flags, caveats for any retools/DX-script/dumpinfo tool | `.claude/references/tool-catalog.md` |
+| Whether a task should run inline vs be delegated | `.claude/rules/tool-dispatch.md` |
+| Bootstrap ordering, parallel dual-backend runs, delegation table | `.claude/rules/subagent-workflow.md` |
+| Attaching to a live process, breakpoints, tracing, memory patching | `/dynamic-analysis` skill (main agent only — this agent must not use livetools) |
+| Porting a DX9 game to FFP for RTX Remix (renderer.cpp, ffp_state, vertex decls, skinning) | `dx9-ffp-port` skill |
+| D3D9-specific static questions (VS/PS constants, render states, vertex formats) | DX analysis scripts (`rtx_remix_tools/dx/scripts/`) before general retools |