idasql-inspired analysis layer: SQLite index, SQL query, Ghidra daemon

[Night1099]

Summary

Adds an idasql-inspired analysis layer to the RE toolkit: a per-game SQLite
index, a read-only SQL front-end, a consolidated kb.h parser, and a warm
per-project Ghidra daemon with transparent routing. Static findings that
used to live only in a flat kb.h are now also queryable as structured data,
and repeat Ghidra work (decompile / export / kb-apply) skips the ~3s JVM cold
start.

What's new

retools/index.py — per-game SQLite index (GameIndex)

• Schema for funcs/names/xrefs/strings/imports/entries/segments/blocks, plus
  callers / callees / grep views.
• WAL-with-DELETE fallback, schema-version guard, bit-63 address guard.
• Source-priority upsert: authoritative Ghidra rows overwrite provisional
  bootstrap rows, and re-running bootstrap can never downgrade them back.
• index status <Game> CLI.

retools/query.py — read-only SQL front-end

• query <Game> "SELECT …" over index.db via a file:…?mode=ro connection
  (can never mutate the index). Text + --json envelope, --list-tables,
  --schema TABLE.

retools/kb.py — one kb.h parser

• Single source of truth for the kb.h grammar; the three former duplicate
  parsers (decompiler, context, bootstrap) now delegate to it.

retools/pyghidra_backend.py — export + kb-apply

• export seeds index.db with authoritative source='ghidra'
  funcs/names/xrefs/blocks.
• kb-apply pushes kb.h names/labels/prototypes into the Ghidra project.
• All Ghidra imports are lazy so the module loads with no JDK.

retools/ghidra_server.py + ghidra_client.py — per-project daemon

• Warm Ghidra program on port 27043 (livetools owns 27042). decompile,
  export, and kb-apply route through a live daemon automatically;
  RETOOLS_GHIDRA_COLD=1 / --cold forces cold. Tracks itself in
  patches/<Game>/ghidra/.state.json.

bootstrap.py / context.py

• Bootstrap seeds index.db (source='bootstrap') from pefile data.
• context assemble resolves callees from index.db when present, falling
  back to the disassembly scan otherwise.

Step	Result
bootstrap	seeds index.db (9,648 funcs, 35,201 strings, 376 imports) + kb.h
Ghidra analyze	~96s, reusable project
export (via daemon)	funcs=10,243, names=10,243, xrefs=220,237, blocks=137,368
source-priority	funcs = 12,502 = 10,243 Ghidra + 2,258 surviving bootstrap (no downgrade)
warm decompile	0.46s → 0.15s (cached DecompInterface) vs 3.3s cold
kb-apply	2,890 functions renamed from kb.h, persisted + re-exported
context assemble	real named callee edges from the index

master vs this branch, same task: master reaches the same answer but with
16 modules (no index/query/ghidra_server), a bootstrap that produces no
index.db, ~4s cold decompiles with no warm path, and context callees
limited to edx: indirect call placeholders. The improvements don't change the
answer — they make the toolkit queryable and fast to iterate (SQL over
220k xrefs, 0.15s warm decompiles, named cross-references).

Review findings fixed

Correctness

• Ghidra daemon cross-project routing hardened: pid-first liveness check +
  server-side game-identity handshake + dropped SO_REUSEADDR.
• export default --db derives from --project (was the binary stem →
  wrong dir); _iter_blocks keys blocks by containing-function entry.
• _route_daemon propagates socket.timeout instead of silently cold-retrying
  into a mid-operation daemon; sends absolute paths (cwd-independent).
• context.assemble: verifies the start is a known function before trusting an
  empty callee result, falls back on a schemaless index.db (was crashing),
  recovers indirect-call markers, and prefers fresh kb.h names.
• index.db path anchoring unified; schema-version repaired on interrupted
  first open; decompiler.py script-mode import fixed.
• kb-apply transaction bug (found on real Ghidra): removed the manual
  program.save() that fails with "Unable to lock due to active transaction"
  inside pyghidra's open_program context — persist via project.save on exit
  (cold) / program close (daemon); @ entries at data addresses become labels,
  not bogus functions.

Cleanup / conventions

• ix_xrefs_from_func index; narrowed catch-all excepts; single-threaded daemon
  loop (removes the lock, close-variant split, thread list, idle self-connect);
  reused framing; error response on malformed frames; cached DecompInterface;
  parse kb.h once per assemble; shared resolve_db; one bootstrap string
  sweep; extracted shared _kb_apply_txn.

Code Reveiw ran and done CODE_REVIEW.md

[Kim2091]

Can test later today

[Kim2091]

Review + follow-up fix

Transparency: most of this review was produced autonomously by Claude Fable 5 (running in Claude Code). I took over for the hands-on part — reproducing the issue, writing the fix + test, and pushing it to this branch.

What the PR does

Adds an idasql-inspired analysis layer: a per-game SQLite index (index.py), a read-only SQL front-end (query.py), a consolidated kb.h parser (kb.py), export + kb-apply in pyghidra_backend.py, and a warm per-project Ghidra daemon (ghidra_server.py / ghidra_client.py) with transparent routing. Also folds the duplicated per-harness agent config into a single AGENTS.md.

Verification run

• Full test suite: 244 passed, 1 skipped.
• End-to-end smoke test on a real analyzed PE (bootstrap → index/query → Ghidra analyze → cold vs. warm decompile → export → kb-apply → context assemble):
  • warm daemon decompile ~0.28s vs ~7.9s cold — matches the headline claim
  • source-priority merge holds: Ghidra rows win, provisional bootstrap rows survive where Ghidra has none, and a bootstrap re-run doesn't downgrade them
  • kb-apply renames persist; a re-export reflects them
  • the read-only query connection correctly rejects writes
  • the wrong-project handshake rejects cross-project routing

Solid overall — the core is functional and the performance win is real.

Findings

1. [Medium — fixed in 2af273a, now on this branch] --cold crashed when the project's daemon was live.
Running decompile/export/kb-apply with --cold (or RETOOLS_GHIDRA_COLD=1) while that project's daemon was up died with a raw LockException traceback — the Ghidra project lock is exclusive, so the in-process open can't acquire it — instead of the [error] … string every other path returns. _route_daemon now checks daemon liveness before the cold branch and returns a clean error when the daemon holds the lock; with no live daemon it falls through to the cold path exactly as before. Also corrected the --cold help text (it can't "bypass any live daemon" for the same project) and replaced the single cold-mode test with two covering both branches. Verified end-to-end: cold-while-daemon-live → clean error, and both warm routing and cold-without-daemon still work.

2. [Low, not fixed] Daemon .state.json binary field is always null — it's written once at startup before any program opens and never updated afterward. Harmless (routing keys off game/pid/port), but the docs list binary as tracked. Worth a small follow-up.

Minor notes: the single-connection daemon uses a 300s socket timeout, so a stalled client blocks others for up to 5 min (fine for local single-user use); query.py runs arbitrary agent-generated SQL (read-only mode blocks writes, but ATTACH/PRAGMA remain reachable — no real risk in a local single-operator tool).

#2 and the minor notes are non-blocking.