diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index a929532f..80421f6b 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -1,18 +1,10 @@ # Vibe Reverse Engineering -- Claude Code Instructions -## Read-Only Templates +Shared conventions (project overview, read-only templates, workspace/backup/KB rules, engineering standards, code comments) are canonical in the root file, auto-loaded here: -These directories are **shared tooling and templates**. Do not modify them for game-specific work — per-game changes go in `patches//`. +@../AGENTS.md -- `rtx_remix_tools/dx/remix-comp-proxy/` — proxy framework **template** (copied per-game) -- `rtx_remix_tools/dx/scripts/` — DX9 analysis scripts (shared tooling) -- `retools/` — static analysis toolkit (shared tooling) -- `livetools/` — Frida-based dynamic analysis (shared tooling) -- `graphics/` — DX9 tracer framework (shared tooling) - -**Per-game work goes in `patches//`.** When starting a new game, copy `rtx_remix_tools/dx/remix-comp-proxy/` (excluding `build/`) to `patches//` and edit the copy. If the user says "edit remix-comp-proxy code" without specifying, ask whether they mean the template or a game copy. - -Shared tooling can be modified to improve the tools themselves — just not for game-specific customization. +The sections below are Claude Code-specific. --- @@ -30,58 +22,6 @@ The main agent owns `livetools` — always use them to verify static findings, p --- -## Engineering Standards - -Every change should make the codebase better, not just make the problem go away. If a solution needs a paragraph to justify why it's not a hack, it's a hack. - -### Remove -- **Fixes in the wrong layer**: a guard on a canvas to suppress commits that a model should own. Put the fix where the problem originates. -- **Tolerance inflation**: widening deltas or adding retries to hide flaky behavior. If the value is wrong, find out why. -- **Catch-all exception swallowing**: `try/except Exception: pass` to hide symptoms. -- **Excessive error/null handling**: adding too many error/None "if" checks. If the error is expected, handle it. If unexpected, raise it. -- **God methods**: 200+ line functions doing multiple things. Break into named steps. Focus on cognitive load. Design for fewer indentation levels. -- **Leaky abstractions**: implementation details leaking into layers/modules that should be agnostic of one another. - -### Design For -- **Single responsibility**: one component, one job. If you need "and" to describe it, split it. -- **Ownership**: the component that creates the problem owns the fix. -- **Minimal public surface**: expose what consumers need, nothing more. - -### Commit to the New Code -- **No legacy fallbacks**: if you replace a system, remove the old one. -- **No dead code**: commented-out blocks, unused imports, orphan functions "just in case". Version control is the safety net. -- **No multiple paths to the same result**: one way to do each thing. If two paths exist, one is wrong. -- **No half-migrations**: finish the job -- update every reference, remove old APIs. - -### Smell Tests -- "It works if I add a sleep" -- broken data flow. -- "It works if I read from widget instead of storage" -- the two are out of sync. -- "It passes alone but fails with other tests" -- shared mutable state leaking. -- "I added a flag to skip this code path" -- why does that path run in the first place? - -## Code Comments - -Each file reads as if it was always designed this way. Comments guide the next developer, not narrate the development journey. - -### Remove -- **Implementation backstories**: "We do this because the other day X happened" -- **Obvious narration**: "Create the attribute", "Loop through keys", "Check if valid" -- if the code says it, the comment is noise -- **Debugging breadcrumbs**: "Without this, subsequent tests may see the modifier key as still held" -- **Trial-and-error reasoning**: "We tried X but it caused Y so we do Z instead" - -### Keep -- **Non-obvious design decisions**: stated as *what* and *why this design*, not *what happened to us* -- **Tricky invariants**: conditions that would be easy to accidentally break -- **API contracts**: docstrings on public methods with Args, Returns, Raises - -### Prefer Instead -- **Rename** a variable or function to be self-explanatory rather than adding a comment -- **Docstrings** on classes and public methods (Google style: `Args:`, `Returns:`, `Raises:`) -- **Type hints** over comments about expected types -- **Short inline comments** on the *why*, never the *what* - ---- - ## DX9 FFP Porting Invoke the **`dx9-ffp-port` skill** before editing `renderer.cpp`, `ffp_state.cpp`, `remix-comp-proxy.ini`, or draw routing; porting a game for RTX Remix; diagnosing VS constants, vertex declarations, matrix mapping, or skinning; or building/deploying a remix-comp-proxy patch. diff --git a/.claude/agents/static-analyzer.md b/.claude/agents/static-analyzer.md index 6c855591..94d131ce 100644 --- a/.claude/agents/static-analyzer.md +++ b/.claude/agents/static-analyzer.md @@ -27,7 +27,7 @@ test -f retools/data/signatures.db || python retools/sigdb.py pull ```bash grep -cE '^[@$]|^struct |^enum ' patches//kb.h 2>/dev/null || echo 0 ``` -If the count is under 50 (or the file doesn't exist), run `python -m retools.bootstrap --project ` first. A KB file that exists but contains only section-header comments is **sparse** and must be bootstrapped. Do not skip bootstrap just because the file exists. +If the count is under 50 (or the file doesn't exist), run `python -m retools.bootstrap --project patches/` first (`--project` is the output directory path, so it must include the `patches/` prefix — a bare name writes kb.h outside the project tree). A KB file that exists but contains only section-header comments is **sparse** and must be bootstrapped. Do not skip bootstrap just because the file exists. **4. Ghidra project**: Check if a Ghidra project exists for the binary: ```bash @@ -35,30 +35,58 @@ python retools/pyghidra_backend.py status --project patches/ ``` If "Not analyzed", run `python retools/pyghidra_backend.py analyze --project patches/`. Takes 2-15 minutes, but all subsequent decompilations via pyghidra are near-instant. +**5. Index**: Check whether the project has an index.db and what's in it before scanning the binary yourself: +```bash +python -m retools.index status +``` +If `funcs`/`xrefs` show `source='bootstrap'` only (or the table is empty), and a Ghidra project exists, run `pyghidra_backend.py export` to seed authoritative facts — see "Query-first workflow" below. + ## Running Tools Run all tools from the repo root. Use `python -m retools.` or `python retools/.py` syntax: -### Decompilation (two backends) +### Decompilation -- Ghidra primary, r2ghidra fallback -**pyghidra (preferred when Ghidra project exists)** — better MSVC type propagation, library call resolution, larger function scope detection: +**pyghidra is the primary backend** once a Ghidra project exists — better MSVC type propagation, library call resolution, larger function scope detection, and its facts can be exported into `index.db` for instant SQL lookups later: ``` python retools/pyghidra_backend.py decompile binary.exe 0x401000 --project patches/proj ``` -**r2ghidra (fast fallback)** — better `__thiscall` on small functions, no JVM startup: +**r2ghidra is the zero-setup fallback and second opinion** — no Ghidra install required, better `__thiscall` recovery on small functions, no JVM startup, and useful to cross-check a pyghidra decompile that looks wrong: ``` python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h --backend pdg ``` -**Auto mode (tries pyghidra first, falls back to r2ghidra)**: +**Auto mode (tries pyghidra first, falls back to r2ghidra)** — routing unchanged: ``` python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h --project patches/proj ``` When told to use a specific backend, use it. Otherwise prefer auto mode with both `--types` and `--project`. +**Ghidra daemon**: if `python -m retools.ghidra_server ` is running (port 27043; livetools owns 27042), `decompile`/`export`/`kb-apply` route through it automatically and repeat calls become sub-second. Warm the server yourself before a batch of decompiles on the same project: `python -m retools.ghidra_server --idle 600` (background it). `RETOOLS_GHIDRA_COLD=1` or `--cold` forces a cold in-process run when you need to bypass the daemon. The daemon records its pid/port/project/binary in `patches//ghidra/.state.json`, deleted on shutdown once the Ghidra program is closed and the Windows `.rep` lock is released. + +### Query-first workflow + +Before re-scanning a binary with xrefs/datarefs/search/funcinfo, check whether `index.db` already has the answer — a SQL query against a local file is cheaper than re-disassembling: + +```bash +python -m retools.index status # per-table counts + schema_version +python -m retools.query --list-tables # confirm what's queryable +python -m retools.query --schema funcs # PRAGMA table_info before writing joins +python -m retools.query "SELECT * FROM callers WHERE callee_addr=0x401000" +python -m retools.query "SELECT * FROM grep WHERE name LIKE '%Ground%'" --json +``` + +Only fall back to `xrefs.py`/`datarefs.py`/`search.py`/`funcinfo.py` for facts `index.db` doesn't have yet (e.g. no `export` has run, or the question needs a live disassembly detail not captured by the schema). + +**Hard pushdown rule**: `decompile` and `export` require a specific function address (or, for `export`, an analyzed program) — never invoke them without one, or you decompile/scan the whole binary instead of the function you actually need. If you don't have an address yet, get one from `query`, `search`, or `xrefs` first. + +**Read-First mutation discipline**: `kb-apply` mutates the Ghidra project. Always decompile or `query` the target function first to confirm the current name/prototype, run `kb-apply`, then **re-decompile the same function** to verify the change landed before reporting it as done. `kb-apply` is idempotent — re-running it should produce stable counts and no errors, so if a second run changes anything, treat that as a bug, not expected behavior. + +**Cost guard**: run `export` once per analysis pass (after `kb-apply`, so exported names reflect it), not once per query — repeated `export` calls re-walk the whole program for no benefit once `index.db` is current. + ### Other tools ``` python -m retools.search binary.exe strings -f "error" --xrefs @@ -67,13 +95,18 @@ python -m retools.callgraph binary.exe 0x401000 --up 3 python -m retools.structrefs binary.exe --aggregate --fn 0x401000 --base esi python -m retools.dumpinfo crash.dmp diagnose --binary d3d9.dll python -m retools.throwmap d3d9.dll match --dump crash.dmp -python -m retools.bootstrap binary.exe --project MyGame +python -m retools.bootstrap binary.exe --project patches/MyGame python -m retools.sigdb scan binary.exe --db retools/data/signatures.db python -m retools.sigdb identify binary.exe 0x401000 --db retools/data/signatures.db python -m retools.sigdb fingerprint binary.exe python -m retools.context assemble binary.exe 0x401000 --project MyGame python retools/pyghidra_backend.py analyze binary.exe --project patches/MyGame python retools/pyghidra_backend.py status binary.exe --project patches/MyGame +python retools/pyghidra_backend.py export binary.exe --project patches/MyGame +python retools/pyghidra_backend.py kb-apply binary.exe --project patches/MyGame --kb patches/MyGame/kb.h +python -m retools.index status MyGame +python -m retools.query MyGame "SELECT * FROM funcs WHERE name LIKE '%Update%'" +python -m retools.ghidra_server MyGame --idle 600 ``` If `retools/data/signatures.db` is missing, run `python -m retools.sigdb pull` to download it. @@ -140,3 +173,16 @@ Also update `patches//kb.h` with any new function signatures, structs, In your return message, state the file path you wrote to and give a brief summary. The main agent will read the file for full details. Update your agent memory with significant architectural discoveries, identified subsystems, and class hierarchies that will be useful in future sessions. + +## Routing to Adjacent Skills/Docs + +This agent owns offline static analysis. Hand off to the right reference/skill instead of improvising: + +| Need | Go to | +|------|-------| +| Full tool syntax, flags, caveats for any retools/DX-script/dumpinfo tool | `.claude/references/tool-catalog.md` | +| Whether a task should run inline vs be delegated | `.claude/rules/tool-dispatch.md` | +| Bootstrap ordering, parallel dual-backend runs, delegation table | `.claude/rules/subagent-workflow.md` | +| Attaching to a live process, breakpoints, tracing, memory patching | `/dynamic-analysis` skill (main agent only — this agent must not use livetools) | +| Porting a DX9 game to FFP for RTX Remix (renderer.cpp, ffp_state, vertex decls, skinning) | `dx9-ffp-port` skill | +| D3D9-specific static questions (VS/PS constants, render states, vertex formats) | DX analysis scripts (`rtx_remix_tools/dx/scripts/`) before general retools | diff --git a/.claude/agents/web-researcher.md b/.claude/agents/web-researcher.md deleted file mode 100644 index 786e2337..00000000 --- a/.claude/agents/web-researcher.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -name: web-researcher -description: Web research and documentation lookups. Delegate here for API references, library documentation, SDK docs, file format specs, protocol details, or any question requiring external knowledge. Use instead of doing web research in the main conversation. -disallowedTools: Edit, Write, NotebookEdit, Bash, Agent -model: sonnet ---- - -You are a technical research assistant supporting a reverse engineering workflow. You fetch documentation, API references, and technical specs from the web and return concise, actionable findings. - -## Tools - -- **WebFetch**: Fetch and extract content from a specific URL -- **WebSearch**: Search the web for technical information -- **Context7 MCP**: Use `resolve-library-id` then `query-docs` for library-specific documentation (DirectX, Win32 API, game engine docs, etc.) -- **Read**: Read local files for context about what's being researched - -## How to Work - -1. Understand what the caller needs — a specific API signature, a file format layout, a protocol detail, etc. -2. Search or fetch the most authoritative source (MSDN, official docs, specs) -3. Extract the specific information needed — don't return entire pages -4. Format findings for direct use in reverse engineering or code writing - -## Output - -Return concise, structured results: -- The specific answer or data requested -- Key details (function signatures, struct layouts, enum values, constants) -- Source URL for reference -- Any caveats or version-specific differences - -Do NOT return long summaries or background context unless specifically asked. The caller already knows the domain — they need the specific data point. diff --git a/.claude/references/tool-catalog.md b/.claude/references/tool-catalog.md index ca27fde5..3fd881da 100644 --- a/.claude/references/tool-catalog.md +++ b/.claude/references/tool-catalog.md @@ -27,6 +27,8 @@ These are fast (<5s) and allowed inline: - "Trace where this value comes from" → `python -m retools.dataflow $B $VA --slice TARGET_VA:REG` - "Build an ASI patch DLL" → `python -m retools.asi_patcher build spec.json` - "Does a Ghidra project exist for this binary?" → `python retools/pyghidra_backend.py status $B --project $P` +- "What's in this game's index?" → `python -m retools.index status [--db PATH]` +- "Query facts already indexed (funcs, names, xrefs, strings, imports, callers/callees)" → `python -m retools.query "SQL" [--db PATH] [--json] [--list-tables] [--schema TABLE]` ### Delegate to `static-analyzer` subagent @@ -35,13 +37,13 @@ Everything else. Tell the subagent WHAT you need, not HOW to run it — it has t **D3D9-specific questions?** Check the DX analysis scripts section below first — they're faster and more targeted than general retools for D3D API usage, device calls, shader constants, and vertex formats. - "What does this function do?" → decompile + callgraph + xrefs + dataflow --constants -- "Who calls this function?" → xrefs or callgraph --up -- "What does this function call?" → callgraph --down (add --indirect for vtable calls) +- "Who calls this function?" → xrefs or callgraph --up (prefer `retools.query "SELECT * FROM callers WHERE callee_addr=..."` when index.db exists) +- "What does this function call?" → callgraph --down (add --indirect for vtable calls) (prefer `retools.query "SELECT * FROM callees WHERE caller=..."` when index.db exists) - "Who calls this virtual method?" → xrefs --indirect + filter by vtable slot offset - "What constant reaches this call?" → dataflow --constants or --slice VA:REG - "Resolve a switch/jump table" → cfg (auto-resolves MSVC switch patterns) -- "Find a string and who uses it" → string search with xrefs -- "Where is this global read/written?" → datarefs +- "Find a string and who uses it" → string search with xrefs (prefer `retools.query "SELECT * FROM grep WHERE name LIKE '%...%'"` when index.db exists) +- "Where is this global read/written?" → datarefs (prefer `retools.query` against `names`/`xrefs` when index.db exists) - "Where is struct field +0x54 used?" → structrefs - "What does this struct look like?" → structrefs --aggregate - "What C++ class is this vtable?" → RTTI resolution @@ -51,6 +53,8 @@ Everything else. Tell the subagent WHAT you need, not HOW to run it — it has t - "Map all throw sites to error strings" → throwmap list - "First time analyzing a binary?" → bootstrap (2-5 min) + pyghidra analyze (5-15 min) in parallel - "Bulk signature scan" → sigdb scan (1-3 min) +- "Seed/refresh the index from a Ghidra project" → `pyghidra_backend.py export` (funcs/names/xrefs/blocks, source='ghidra', authoritative over provisional bootstrap rows) +- "Push kb.h names/prototypes into the Ghidra project" → `pyghidra_backend.py kb-apply` (idempotent) - Any combination of the above ### Live tools (main agent, requires attached process) @@ -102,19 +106,24 @@ These are fast first-pass scanners — they surface candidate addresses. Follow | `pyghidra_backend.py analyze $B --project $P` | **Full Ghidra analysis** -- one-time, saves reusable project | `pyghidra_backend.py analyze game.exe --project patches/MyGame` | | `pyghidra_backend.py decompile $B $VA --project $P` | Decompile via saved Ghidra project | `pyghidra_backend.py decompile game.exe 0x401000 --project patches/MyGame` | | `pyghidra_backend.py status $B --project $P` | Check if Ghidra project exists | `pyghidra_backend.py status game.exe --project patches/MyGame` | -| `funcinfo.py $B $VA` | Find function start/end, rets, calling convention, callees | `funcinfo.py binary.exe 0x401000` | +| `pyghidra_backend.py export $B --project $P [--db]` | Seed funcs/names/xrefs/blocks from an analyzed Ghidra program into index.db (`source='ghidra'`, overwrites provisional bootstrap rows at the same address) | `pyghidra_backend.py export game.exe --project patches/MyGame` | +| `pyghidra_backend.py kb-apply $B --project $P --kb $KB` | Push kb.h names/prototypes/globals/typedefs into the Ghidra project. Idempotent — safe to re-run | `pyghidra_backend.py kb-apply game.exe --project patches/MyGame --kb patches/MyGame/kb.h` | +| `index.py status $GAME [--db]` | Per-table row counts + schema_version for `patches//index.db` | `python -m retools.index status MyGame` | +| `query.py $GAME "SQL" [--db] [--json] [--list-tables] [--schema TABLE]` | Read-only SQL over index.db. Views: `callers`, `callees`, `grep`. Addresses render as hex via `printf('0x%x', address)`. Connection is opened read-only — cannot mutate | `python -m retools.query MyGame "SELECT * FROM grep WHERE name LIKE '%Ground%'"` | +| `ghidra_server.py $GAME [--idle SECS]` | Run a per-project Ghidra daemon (port 27043; livetools owns 27042) holding one warm program so repeat `decompile`/`export`/`kb-apply` calls are sub-second instead of paying JVM+analysis startup each time. Tracks itself in `patches//ghidra/.state.json` (pid/port/project/binary), deleted on shutdown once the Ghidra program is closed | `python -m retools.ghidra_server MyGame --idle 600` | +| `funcinfo.py $B $VA` | Find function start/end, rets, calling convention, callees. Prefer `retools.query` on `funcs`/`blocks` when index.db exists | `funcinfo.py binary.exe 0x401000` | | `cfg.py $B $VA` | Control flow graph (basic blocks + edges, text or mermaid). Resolves MSVC switch/jump tables automatically. `--switch-details` shows table info | `cfg.py binary.exe 0x401000 --format mermaid` | | `callgraph.py $B $VA` | Caller/callee tree (multi-level, --up/--down N). `--indirect` adds vtable/fptr calls to --down trees | `callgraph.py binary.exe 0x401000 --down 2 --indirect` | -| `xrefs.py $B $VA` | Find all calls/jumps TO an address. `--indirect` also scans for `call [reg+offset]`, `call [reg]`, `call [addr]` | `xrefs.py binary.exe 0x401000 --indirect` | +| `xrefs.py $B $VA` | Find all calls/jumps TO an address. `--indirect` also scans for `call [reg+offset]`, `call [reg]`, `call [addr]`. Prefer `retools.query "SELECT * FROM callers WHERE callee_addr=..."` when index.db exists | `xrefs.py binary.exe 0x401000 --indirect` | | `dataflow.py $B $VA` | Forward constant propagation (`--constants`) or backward register slice (`--slice VA:REG`) within a function | `dataflow.py binary.exe 0x401000 --constants` | -| `datarefs.py $B $VA` | Find instructions that reference a global address (mem deref + `--imm` for push/mov constants) | `datarefs.py binary.exe 0x7A0000 --imm` | +| `datarefs.py $B $VA` | Find instructions that reference a global address (mem deref + `--imm` for push/mov constants). Prefer `retools.query` against `xrefs`/`names` when index.db exists | `datarefs.py binary.exe 0x7A0000 --imm` | | `structrefs.py $B $OFF` | Find all `[reg+offset]` accesses (struct field usage) | `structrefs.py binary.exe 0x54 --base esi` | | `structrefs.py $B --aggregate` | Reconstruct C struct from all field accesses in a function | `structrefs.py binary.exe --aggregate --fn 0x401000 --base esi` | | `vtable.py $B dump $VA` | Dump C++ vtable slots with instruction preview | `vtable.py binary.exe dump 0x6A0000` | | `vtable.py $B calls $OFF` | Find all indirect `call [reg+offset]` (vtable call sites) | `vtable.py binary.exe calls 0xB0` | | `rtti.py $B vtable $VA` | Resolve C++ class name + inheritance chain from vtable (MSVC RTTI) | `rtti.py binary.dll vtable 0x6A0000` | | `rtti.py $B throwinfo $RVA` | Resolve exception type from `_ThrowInfo` (MSVC RTTI) | `rtti.py binary.dll throwinfo 0x5040CF8` | -| `search.py $B strings` | Extract strings with keyword filter | `search.py binary.exe strings -f render,draw` | +| `search.py $B strings` | Extract strings with keyword filter. Prefer `retools.query "SELECT * FROM grep WHERE name LIKE '%...%'"` when index.db exists | `search.py binary.exe strings -f render,draw` | | `search.py $B strings --xrefs` | Find strings AND code locations that reference them | `search.py binary.exe strings -f "error" --xrefs` | | `search.py $B pattern` | Find exact byte pattern | `search.py binary.exe pattern "D9 56 54 D8 1D"` | | `search.py $B imports` | List PE imports, filter by DLL | `search.py binary.exe imports -d kernel32` | @@ -123,7 +132,7 @@ These are fast first-pass scanners — they surface candidate addresses. Follow | `search.py $B insn --near` | Find instructions near another pattern | `search.py binary.dll insn "mov *,0x10000" --near "cmp *,0x10000" --range 0x400` | | `readmem.py $B $VA $TYPE` | Read typed data (float, uint32, ptr, bytes...) | `readmem.py binary.exe 0x401000 float` | | `asi_patcher.py build` | Generate .asi DLL patch from JSON spec | `asi_patcher.py build spec.json --vcvarsall ...` | -| `bootstrap.py $B --project $P` | Auto-seed KB: compiler ID, signatures, RTTI, imports, propagation | `bootstrap.py game.exe --project Warband` | +| `bootstrap.py $B --project $P` | Auto-seed KB: compiler ID, signatures, RTTI, imports, propagation. `--project` is the **output directory** (`$P` = `patches/`), not a bare name | `bootstrap.py game.exe --project patches/Warband` | | `sigdb.py scan $B` | Bulk signature scan against DB | `sigdb.py scan game.exe` | | `sigdb.py identify $B $VA` | Single function signature lookup (multi-tier) | `sigdb.py identify game.exe 0x401200` | | `sigdb.py fingerprint $B` | Identify compiler version (Rich header + markers + imports) | `sigdb.py fingerprint game.exe` | @@ -175,7 +184,7 @@ A proxy DLL that intercepts all 119 `IDirect3DDevice9` methods, capturing every ``` python -m graphics.directx.dx9.tracer codegen -o d3d9_trace_hooks.inc # C hooks (standalone proxy) -python -m graphics.directx.dx9.tracer codegen -f cpp -o tracer_dispatch.inc # C++ dispatch (remix-comp module) +python -m graphics.directx.dx9.tracer codegen -f cpp -o tracer_dispatch.inc # C++ dispatch (remix-comp-proxy module) cd graphics/directx/dx9/tracer/src && build.bat # build standalone proxy DLL # Deploy d3d9.dll + proxy.ini to game directory python -m graphics.directx.dx9.tracer trigger --game-dir # trigger capture (3s countdown) @@ -289,13 +298,17 @@ Minidumps vary in how much data they capture depending on `MiniDumpWriteDump` fl These tools find references via absolute memory operands, immediate values (with `--imm` flag), and RIP-relative addressing. If you suspect a reference exists but the tool doesn't find it, the address might be computed at runtime. Try `search.py pattern` with the address bytes directly, or use `livetools memwatch`. -### `pyghidra_backend.py` -- requires Ghidra installation +### `pyghidra_backend.py` -- Ghidra primary, r2ghidra fallback + +Ghidra (via `pyghidra_backend.py`, indexed into `index.db`, daemon-backed, kb-applied) is the **primary** decompilation backend once a project exists — it gives better type propagation, library call resolution, and lets `retools.query` answer structural questions without re-scanning the binary. r2ghidra (`decompiler.py --backend pdg`) is the **zero-setup fallback and second opinion**: no Ghidra install required, faster on small functions, and useful to cross-check a pyghidra result that looks wrong. `decompiler.py --backend auto` tries pyghidra first and falls back to r2ghidra automatically — this routing is unchanged. Requires Ghidra 11.x+ installed and `GHIDRA_INSTALL_DIR` environment variable set. **Optional** -- the toolkit works without it (r2ghidra remains the fallback). **Disk usage**: Ghidra projects are ~10-20x the binary size. A 30MB game exe produces a ~300-600MB `.rep/` directory under `patches//ghidra/`. This directory is already covered by `.gitignore` (the `patches/` exclusion). -**First-time analysis** takes 5-15 minutes depending on binary size. Subsequent decompilation from the saved project is instant (<1s plus ~3s JVM startup per process). +**First-time analysis** takes 5-15 minutes depending on binary size. Subsequent decompilation from the saved project is near-instant (<1s plus ~3s JVM startup per cold process) -- or truly sub-second when a `ghidra_server.py` daemon is warm for that project. `export` and `kb-apply` route through the same live daemon when one is running; `RETOOLS_GHIDRA_COLD=1` or `--cold` forces a cold in-process run for either. + +**Index-first**: after `export`, prefer `retools.query` over datarefs/xrefs/search/funcinfo for anything already captured in `index.db` (funcs, names, xrefs, strings, imports, blocks) -- it's a local SQL query instead of a fresh binary scan. Fall back to the scanners only for facts index.db doesn't have yet. ### `livetools` -- static vs runtime addresses diff --git a/.claude/rules/subagent-workflow.md b/.claude/rules/subagent-workflow.md index 0900bf61..6aeb4ba7 100644 --- a/.claude/rules/subagent-workflow.md +++ b/.claude/rules/subagent-workflow.md @@ -4,7 +4,7 @@ description: Subagent delegation rules — when to spawn static-analyzer vs run # Subagent Workflow -Main agent: **live tools**, **dx9tracer capture**, **user interaction**, **synthesis**. Heavy static analysis and web research → subagents. +Main agent: **live tools**, **dx9tracer capture**, **user interaction**, **synthesis**. Heavy static analysis → subagents. ## Pre-flight: Ensure Ghidra Backend @@ -14,7 +14,7 @@ Before first pyghidra use, run `python verify_install.py` — if pyghidra shows When analyzing a binary for the first time (no existing or sparsely populated `patches//kb.h`), **always bootstrap before other static analysis**: -1. Spawn `static-analyzer`: `bootstrap.py --project ` — seeds kb.h with RTTI, CRT/library IDs, compiler info, propagated labels. **2-5 minutes.** After bootstrap, ALL `decompiler.py` calls must use `--types patches//kb.h`. +1. Spawn `static-analyzer`: `bootstrap.py --project patches/` (`--project` is the output directory path — include the `patches/` prefix or kb.h lands outside the project tree) — seeds kb.h with RTTI, CRT/library IDs, compiler info, propagated labels. **2-5 minutes.** After bootstrap, ALL `decompiler.py` calls must use `--types patches//kb.h`. 2. **In parallel**, spawn second `static-analyzer`: `pyghidra_backend.py analyze --project patches/` — full Ghidra analysis, reusable project. **5-15 minutes.** After this, use `--project patches/` so `--backend auto` prefers Ghidra. 3. Other static analysis can run in parallel but output is richer after bootstrap. @@ -27,12 +27,13 @@ CLAUDE.md lists allowlisted fast commands (run directly) and the general delegat | Task | Where | Notes | |------|-------|-------| -| Web research (docs, API refs, specs) | `web-researcher` subagent | | | dx9tracer offline analysis | `static-analyzer` subagent | | -| Subsequent Ghidra decompile | `static-analyzer` subagent | Fast: JVM ~3s + decompile <1s | +| Subsequent Ghidra decompile | `static-analyzer` subagent | Fast: JVM ~3s + decompile <1s, sub-second with a warm `ghidra_server.py` daemon | | sigdb scan / build | `static-analyzer` subagent | scan 1-3 min, build 1-5 min | | Dataflow: constants + backward slice (`dataflow.py`) | Main agent | fast (<5s) | -| KB updates from findings | `static-analyzer` writes kb.h | main agent may refine | +| KB updates from findings | `static-analyzer` writes kb.h, then `kb-apply` to push into Ghidra | main agent may refine | +| `index status` / `query` (SQL over index.db) | Main agent | fast (<5s); prefer over xrefs/datarefs/search/funcinfo when index.db already has the answer | +| `pyghidra_backend.py export` (seed index.db from Ghidra) | `static-analyzer` subagent | run once per analysis pass, after `kb-apply` | ## Subagent Output @@ -49,12 +50,14 @@ Multiple `static-analyzer` instances can run in parallel for independent questio ## Dual-Backend Deep Analysis -For complex exploratory tasks (finding subsystems, mapping pipelines), spawn **two parallel agents**: +Ghidra (indexed, daemon-backed, kb-applied) is the primary backend once a project exists — prefer it plus `retools.query` over spawning two agents. Reserve the dual-agent pattern below for two specific cases: **no Ghidra project exists yet** (so there's no `index.db` or warm daemon to lean on), or **pyghidra output on a specific function looks wrong** and you need an independent r2ghidra read to cross-check it. + +When one of those applies, spawn **two parallel agents**: 1. **r2ghidra**: `--backend pdg --types kb.h` → writes `findings_r2.md` 2. **pyghidra**: `pyghidra_backend.py decompile` → writes `findings.md` -r2ghidra: better `__thiscall` recovery, low-level D3D. pyghidra: better library call resolution, type propagation. Merge both for complete picture. Not needed for single-function decompilation — use `--backend auto`. +r2ghidra: better `__thiscall` recovery, low-level D3D, no JVM/project dependency. pyghidra: better library call resolution, type propagation, and its output is exportable into `index.db` for future queries. Merge both for complete picture. Not needed for single-function decompilation once a Ghidra project exists — use `--backend auto` (Ghidra primary, r2ghidra fallback). ## Main Agent During Analysis @@ -66,7 +69,7 @@ r2ghidra: better `__thiscall` recovery, low-level D3D. pyghidra: better library ## Examples **"Analyze game.exe for the first time"** -1. Background: `bootstrap.py game.exe --project MyGame` +1. Background: `bootstrap.py game.exe --project patches/MyGame` 2. Background: `pyghidra_backend.py analyze game.exe --project patches/MyGame` 3. Tell user, run `sigdb.py fingerprint` inline while waiting 4. When both return, all subsequent decompilations use `--types kb.h --project patches/MyGame` diff --git a/.claude/rules/tool-dispatch.md b/.claude/rules/tool-dispatch.md index 72f8521e..74016d8f 100644 --- a/.claude/rules/tool-dispatch.md +++ b/.claude/rules/tool-dispatch.md @@ -19,15 +19,20 @@ Run all tools from repo root via `python -m `. **ALWAYS pass `--types pa - `python -m retools.dataflow $B $VA --slice TARGET_VA:REG` — backward register slice - `python -m retools.asi_patcher build spec.json` — build ASI patch DLL - `python retools/pyghidra_backend.py status $B --project $P` — Ghidra project existence check +- `python -m retools.index status [--db PATH]` — per-table row counts + schema_version for the game's index.db +- `python -m retools.query "SQL" [--db PATH] [--json] [--list-tables] [--schema TABLE]` — read-only SQL over index.db (`callers`/`callees`/`grep` views); prefer this over a fresh xrefs/datarefs/search scan whenever index.db already has the data ## Delegate to `static-analyzer` Everything else in `retools`. Tell it WHAT you need, not HOW. D3D9-specific questions — try DX scripts first (faster). -- Decompile / callgraph / xrefs / string search / datarefs / structrefs / RTTI / throwmap / dumpinfo +- Decompile / callgraph / xrefs / string search / datarefs / structrefs / RTTI / throwmap / dumpinfo — check `index status` / `query` first; only fall back to these scanners when index.db lacks the answer - Bootstrap new binary (2-5 min) / pyghidra analyze (5-15 min) / bulk sigdb scan (1-3 min) +- `pyghidra_backend.py export` (seed index.db from a Ghidra project) / `kb-apply` (push kb.h into the Ghidra project) - dx9tracer offline analysis (summary, render-passes, shader-map, etc.) +**Ghidra daemon**: `python -m retools.ghidra_server [--idle 600]` keeps one warm Ghidra program per project on port 27043 (livetools owns 27042). `decompile`/`export`/`kb-apply` route through a live daemon automatically when one is running for that project — repeat decompiles become sub-second instead of paying JVM startup each time. `RETOOLS_GHIDRA_COLD=1` or `--cold` forces a cold in-process run. The daemon tracks itself in `patches//ghidra/.state.json` (pid/port/project/binary), deleted on shutdown once the Ghidra program is closed. + ## Live tools (main agent, attached process) Full syntax and recipes: the `/dynamic-analysis` skill (canonical livetools reference). diff --git a/.cursor/agents/static-analyzer.md b/.cursor/agents/static-analyzer.md index 8f2acbc3..410b957b 100644 --- a/.cursor/agents/static-analyzer.md +++ b/.cursor/agents/static-analyzer.md @@ -1,141 +1,10 @@ --- name: static-analyzer -description: Offline PE binary analysis using retools. Dispatch this subagent for decompilation, disassembly, xrefs, string/pattern search, struct reconstruction, callgraphs, vtable/RTTI resolution, crash dump analysis, bootstrapping new binaries, signature DB operations, context assembly, and any static analysis task. Use proactively whenever static analysis is needed. +description: Offline PE binary analysis using retools. Dispatch this subagent for decompilation, disassembly, xrefs, string/pattern search, struct reconstruction, callgraphs, vtable/RTTI resolution, crash dump analysis, bootstrapping new binaries, signature DB operations, context assembly, and any static analysis task. Use instead of running retools commands in the main conversation. model: inherit readonly: false --- -You are a reverse engineering analyst specializing in static analysis of PE binaries (.exe and .dll). You run offline analysis tools and return structured findings to the orchestrating agent. +The canonical definition of this agent lives in `.claude/agents/static-analyzer.md`. -## Setup - -On first invocation, read the full tool catalog at `.cursor/rules/tool-catalog.mdc` in the working directory. It contains exact syntax, flags, and caveats for every tool. - -## Pre-flight Checks - -Before any analysis, run these checks in order: - -**1. Verify install**: Run `python verify_install.py` on first invocation. If pyghidra/Ghidra/Java show as WARN, run `python verify_install.py --setup` to auto-download JDK 21 + Ghidra + pyghidra. One-time ~600MB download. - -**2. Signature DB**: If `retools/data/signatures.db` does not exist, pull it first: -```bash -test -f retools/data/signatures.db || python retools/sigdb.py pull -``` - -**3. Bootstrap**: Check if the project KB needs bootstrapping: -```bash -grep -cE '^[@$]|^struct |^enum ' patches//kb.h 2>/dev/null || echo 0 -``` -If the count is under 50 (or the file doesn't exist), run `python -m retools.bootstrap --project ` first. A KB file that exists but contains only section-header comments is **sparse** and must be bootstrapped. Do not skip bootstrap just because the file exists. - -**4. Ghidra project**: Check if a Ghidra project exists for the binary: -```bash -python retools/pyghidra_backend.py status --project patches/ -``` -If "Not analyzed", run `python retools/pyghidra_backend.py analyze --project patches/`. Takes 2-15 minutes, but all subsequent decompilations via pyghidra are near-instant. - -## Running Tools - -Run all tools from the repo root. Use `python -m retools.` or `python retools/.py` syntax: - -### Decompilation (two backends) - -**pyghidra (preferred when Ghidra project exists)** — better MSVC type propagation, library call resolution, larger function scope detection: -``` -python retools/pyghidra_backend.py decompile binary.exe 0x401000 --project patches/proj -``` - -**r2ghidra (fast fallback)** — better `__thiscall` on small functions, no JVM startup: -``` -python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h -python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h --backend pdg -``` - -**Auto mode (tries pyghidra first, falls back to r2ghidra)**: -``` -python -m retools.decompiler binary.exe 0x401000 --types patches/proj/kb.h --project patches/proj -``` - -When told to use a specific backend, use it. Otherwise prefer auto mode with both `--types` and `--project`. - -### Other tools -``` -python -m retools.search binary.exe strings -f "error" --xrefs -python -m retools.xrefs binary.exe 0x401000 -t call -python -m retools.callgraph binary.exe 0x401000 --up 3 -python -m retools.structrefs binary.exe --aggregate --fn 0x401000 --base esi -python -m retools.dumpinfo crash.dmp diagnose --binary d3d9.dll -python -m retools.throwmap d3d9.dll match --dump crash.dmp -python -m retools.bootstrap binary.exe --project MyGame -python -m retools.sigdb scan binary.exe --db retools/data/signatures.db -python -m retools.sigdb identify binary.exe 0x401000 --db retools/data/signatures.db -python -m retools.sigdb fingerprint binary.exe -python -m retools.context assemble binary.exe 0x401000 --project MyGame -python retools/pyghidra_backend.py analyze binary.exe --project patches/MyGame -python retools/pyghidra_backend.py status binary.exe --project patches/MyGame -``` - -If `retools/data/signatures.db` is missing, run `python -m retools.sigdb pull` to download it. - -Collect MORE information per command run. Prefer wide queries over narrow ones — a single decompilation with `--types` is better than five disassembly snippets. - -Always pass `--types ` to `decompiler.py` when a KB file exists for the project. - -## Knowledge Base - -When you discover something significant, update the project KB file (`patches//kb.h`). - -Format: -```c -// Structs, enums, typedefs — no prefix -struct Foo { int x; float y; }; -enum Mode { MODE_A=0, MODE_B=1 }; - -// Function signatures — @ prefix -@ 0x401000 void __cdecl ProcessInput(int key); - -// Global variables — $ prefix -$ 0x7C5548 Object* g_mainObject -``` - -Update KB when you: identify a function's purpose, reconstruct a struct, identify a global, find magic constants, or resolve RTTI class names. - -## What NOT to Do - -- Do NOT use `livetools` commands — those require a live process and are handled by the main agent -- Do NOT use `graphics.directx.dx9.tracer` — capture and trigger are handled by the main agent -- Do NOT edit source code files — only update KB files and write analysis notes to `patches/` - -## Output - -Write findings to the appropriate file, creating it if needed. Append — do not overwrite previous findings. - -- **Default**: `patches//findings.md` -- **If told to use r2ghidra for a dual-backend comparison**: `patches//findings_r2.md` - -Use clear headings per analysis task so the main agent can read specific sections. - -Format: -```markdown -## — - -### Summary - - -### Key Addresses -| Address | Description | -|---------|-------------| -| 0x401000 | FunctionName — what it does | - -### Details - - -### Suggested Live Verification - -``` - -Also update `patches//kb.h` with any new function signatures, structs, or globals discovered. - -In your return message, state the file path you wrote to and give a brief summary. The main agent will read the file for full details. - -Update your agent memory with significant architectural discoveries, identified subsystems, and class hierarchies that will be useful in future sessions. +On invocation, read that file and follow everything below its frontmatter as your instructions: setup, pre-flight checks, tool syntax, query-first workflow, knowledge-base rules, what NOT to do, and the findings output format. All paths it references (`.claude/references/tool-catalog.md`, `.claude/rules/*`) are harness-agnostic and apply here unchanged. diff --git a/.cursor/agents/web-researcher.md b/.cursor/agents/web-researcher.md deleted file mode 100644 index a40c7c8e..00000000 --- a/.cursor/agents/web-researcher.md +++ /dev/null @@ -1,32 +0,0 @@ ---- -name: web-researcher -description: Web research and documentation lookups. Dispatch this subagent for API references, library documentation, SDK docs, file format specs, protocol details, or any question requiring external knowledge. Use proactively when external docs are needed. -model: inherit -readonly: true ---- - -You are a technical research assistant supporting a reverse engineering workflow. You fetch documentation, API references, and technical specs from the web and return concise, actionable findings. - -## Tools - -- **WebFetch**: Fetch and extract content from a specific URL -- **WebSearch**: Search the web for technical information -- **Context7 MCP**: Use `resolve-library-id` then `query-docs` for library-specific documentation (DirectX, Win32 API, game engine docs, etc.) -- **Read**: Read local files for context about what's being researched - -## How to Work - -1. Understand what the caller needs — a specific API signature, a file format layout, a protocol detail, etc. -2. Search or fetch the most authoritative source (MSDN, official docs, specs) -3. Extract the specific information needed — don't return entire pages -4. Format findings for direct use in reverse engineering or code writing - -## Output - -Return concise, structured results: -- The specific answer or data requested -- Key details (function signatures, struct layouts, enum values, constants) -- Source URL for reference -- Any caveats or version-specific differences - -Do NOT return long summaries or background context unless specifically asked. The caller already knows the domain — they need the specific data point. diff --git a/.cursor/rules/code-comments.mdc b/.cursor/rules/code-comments.mdc deleted file mode 100644 index 02315387..00000000 --- a/.cursor/rules/code-comments.mdc +++ /dev/null @@ -1,32 +0,0 @@ ---- -description: Code commenting principles -- what to write, what to remove, when code should speak for itself -alwaysApply: true ---- - -# Code Comments - -## Principle - -Each file reads as if it was always designed this way. Comments guide the next developer, not narrate the development journey. - -Note: These rules are not exhaustive. Extrapolate from the principles and examples to the specific context you are working in. - -## Remove - -- **Implementation backstories**: "We do this because the other day X happened" -- **Obvious narration**: "Create the attribute", "Loop through keys", "Check if valid" -- if the code says it, the comment is noise -- **Debugging breadcrumbs**: "Without this, subsequent tests may see the modifier key as still held" -- **Trial-and-error reasoning**: "We tried X but it caused Y so we do Z instead" - -## Keep - -- **Non-obvious design decisions**: stated as *what* and *why this design*, not *what happened to us* -- **Tricky invariants**: conditions that would be easy to accidentally break -- **API contracts**: docstrings on public methods with Args, Returns, Raises - -## Prefer Instead - -- **Rename** a variable or function to be self-explanatory rather than adding a comment -- **Docstrings** on classes and public methods (Google style: `Args:`, `Returns:`, `Raises:`) -- **Type hints** over comments about expected types -- **Short inline comments** on the *why*, never the *what* diff --git a/.cursor/rules/dx9-ffp-port.mdc b/.cursor/rules/dx9-ffp-port.mdc deleted file mode 100644 index 1c006da1..00000000 --- a/.cursor/rules/dx9-ffp-port.mdc +++ /dev/null @@ -1,281 +0,0 @@ ---- -description: DX9 FFP Proxy porting guide for RTX Remix compatibility. Use when porting a DX9 shader-based game to the fixed-function pipeline. -alwaysApply: false ---- - -# DX9 FFP Proxy — Game Porting Guide - -You are helping a user port a DX9 shader-based game to the fixed-function pipeline. Each game folder under `patches//` is a self-contained remix-comp-proxy project (copied from the template at `rtx_remix_tools/dx/remix-comp-proxy/`). The goal is RTX Remix compatibility: Remix requires FFP geometry to inject path-traced lighting and replaceable assets. Also use the Vibe RE tools (retools, livetools) for static and dynamic analysis to assist with developing this wrapper. They are meant to be used together. - -**SKINNING IS OFF BY DEFAULT.** Do NOT enable skinning, modify skinning code, or discuss skinning infrastructure unless the user explicitly asks for character model / bone / skeletal animation support. Until then, treat skinning as non-existent. When the user does request it, read `src/comp/modules/skinning.hpp` and `src/comp/modules/skinning.cpp` for the full implementation. - -**SKINNING APPROACH: FFP indexed vertex blending, NOT CPU matrix math.** When skinning is enabled, keep BLENDINDICES and BLENDWEIGHT in the vertex declaration and buffer, upload bone matrices via `SetTransform(D3DTS_WORLDMATRIX(n), &boneMatrix[n])`, enable `D3DRS_INDEXEDVERTEXBLENDENABLE = TRUE`, and set `D3DRS_VERTEXBLEND` to the weight count. CPU-side vertex skinning is a **last resort** -- it is extremely expensive and tanks frame rate. Always prefer the hardware path. - ---- - -## What remix-comp-proxy Does - -Each game's remix-comp-proxy folder is a C++20 compatibility mod based on remix-comp-base that intercepts `IDirect3DDevice9` and: - -1. Captures vertex shader constants (View, Projection, World matrices) from `SetVertexShaderConstantF` -2. Parses `SetVertexDeclaration` to detect per-element attributes: BLENDWEIGHT+BLENDINDICES (skinned), POSITIONT (screen-space), NORMAL presence, and per-element byte offsets and types -3. Routes `DrawIndexedPrimitive` by vertex layout: - - No NORMAL -> HUD/UI pass-through (uses different VS constant layout than world geometry) - - Skinned with skinning module enabled -> FFP indexed vertex blending - - Rigid 3D (has NORMAL) -> NULLs shaders, applies FFP transforms, draws -4. Routes `DrawPrimitive` by declaration state: world-space draws (have decl, no POSITIONT, not skinned) engage FFP; screen-space and no-decl draws pass through -5. Applies captured matrices via `SetTransform` (FFP) -6. Sets up texture stages and lighting for FFP rendering (stages 1-7 disabled to prevent stale auxiliary textures reaching Remix) -7. Chain-loads RTX Remix (`d3d9_remix.dll`) - -## Source File Map - -| File | Role | -|------|------| -| `src/comp/main.cpp` | DLL entry, module loading, initialization | -| `src/comp/modules/renderer.cpp` | Draw call routing -- `on_draw_indexed_prim()` and `on_draw_primitive()` | -| `src/comp/modules/renderer.hpp` | Renderer class, `drawcall_mod_context` for save/restore state | -| `src/comp/modules/d3d9ex.cpp` | `IDirect3DDevice9` hook layer -- intercepts all 119 methods | -| `src/comp/modules/d3d9ex.hpp` | D3D9 hook declarations | -| `src/comp/modules/skinning.cpp` | Skinning module (vertex expansion, bone upload, FFP blending) | -| `src/comp/modules/skinning.hpp` | Skinning class interface | -| `src/comp/modules/diagnostics.cpp` | Diagnostic logging to `rtx_comp/diagnostics.log` | -| `src/comp/modules/diagnostics.hpp` | Diagnostics class interface | -| `src/comp/modules/imgui.cpp` | ImGui debug overlay (F4 toggle) | -| `src/shared/common/ffp_state.cpp` | FFP state tracker -- engage/disengage, matrix transforms, texture stages | -| `src/shared/common/ffp_state.hpp` | `ffp_state` class with all state accessors | -| `src/shared/common/config.cpp` | INI config parser for `remix-comp-proxy.ini` | -| `src/shared/common/config.hpp` | Config structures: `ffp_settings`, `skinning_settings`, etc. | -| `remix-comp-proxy.ini` (in `assets/`) | Runtime config: `[FFP]`, `[Skinning]`, `[Diagnostics]`, `[Remix]`, `[Chain]` | -| `build.bat` | Build script: outputs d3d9.dll proxy | - -The codebase is C++20 with a `build.bat` build script, component module system for extensibility. - -## What Needs to Change Per Game - -The VS constant register layout is defined in `src/shared/common/ffp_state.hpp` as member defaults. Edit these when porting, then rebuild: - -```cpp -int vs_reg_view_start_ = 0; int vs_reg_view_end_ = 4; -int vs_reg_proj_start_ = 4; int vs_reg_proj_end_ = 8; -int vs_reg_world_start_ = 16; int vs_reg_world_end_ = 20; -int vs_reg_bone_threshold_ = 20; // first register treated as bone palette -int vs_regs_per_bone_ = 3; // 3 = 4x3 packed, 4 = full 4x4 -int vs_bone_min_regs_ = 3; // min count to qualify as bone upload -``` - -**Bone config:** Run `find_skinning.py` to determine bone start register and upload pattern. Some games upload all bones at once; others upload in groups until hitting a max (e.g., groups of 15, max 75). If grouped, lower `vs_bone_min_regs_`. If bone uploads overlap with non-bone constants, raise `vs_reg_bone_threshold_`. - -Beyond the INI config, users may need to modify: -- `renderer.cpp` `on_draw_indexed_prim()` -- draw call routing (which draws get FFP vs shader pass-through) -- `renderer.cpp` `on_draw_primitive()` -- UI/particle handling -- `ffp_state.cpp` `setup_lighting()`, `setup_texture_stages()`, `apply_transforms()` -- FFP render state and matrix configuration -- `AlbedoStage` in `remix-comp-proxy.ini` `[FFP]` section -- which texture stage holds the diffuse/albedo - -## Porting Workflow - -Follow these steps in order for ideal results. Each step depends on the previous. Be sure to use the Vibe Reverse Engineering tools (retools, livetools) for static and dynamic analysis as well. You do not need to strictly follow the order laid out here. - -### Step 1a: Static Analysis - -Run the analysis scripts to understand the game's D3D9 usage: - -```bash -python rtx_remix_tools/dx/scripts/find_d3d_calls.py "" -python rtx_remix_tools/dx/scripts/find_vs_constants.py "" -python rtx_remix_tools/dx/scripts/decode_vtx_decls.py "" --scan -python rtx_remix_tools/dx/scripts/find_device_calls.py "" -python rtx_remix_tools/dx/scripts/find_skinning.py "" -python rtx_remix_tools/dx/scripts/find_blend_states.py "" -``` - -Key things to find: -- How the game obtains its D3D device (Direct3DCreate9 call site -> CreateDevice call) -- Which functions call `SetVertexShaderConstantF` and with what register/count patterns -- What vertex declaration formats the game uses (BLENDWEIGHT/BLENDINDICES = skinning) -- Where the main rendering loop/draw calls live - -### Step 1b: D3D9 Frame Trace (recommended -- fastest path to answers) - -Deploy the D3D9 tracer (`graphics/directx/dx9/tracer/bin/`) to the game directory, capture 2 frames, then run analysis. This is the fastest way to answer all three porting questions without manual RE: - -```bash -python -m graphics.directx.dx9.tracer trigger --game-dir -python -m graphics.directx.dx9.tracer analyze --shader-map -python -m graphics.directx.dx9.tracer analyze --const-provenance -python -m graphics.directx.dx9.tracer analyze --vtx-formats -python -m graphics.directx.dx9.tracer analyze --render-passes --pipeline-diagram -``` - -- `--shader-map` -- CTAB disassembly shows named parameters and register mappings (e.g. `WorldViewProj c0 4`, `WorldView c4 3`, `FogValue c8 1`). Directly reveals which constant registers hold View, Projection, and World matrices. -- `--const-provenance` -- shows which `SetVertexShaderConstantF` call set each register at each draw -- `--vtx-formats` -- groups draws by vertex declaration with full element breakdown (POSITION, NORMAL, BLENDWEIGHT, etc.) -- `--render-passes` + `--pipeline-diagram` -- shows the render pipeline structure and pass types -- `--classify-draws` -- auto-tags draws by render state (alpha, ztest, fog, etc.) - -### Step 2: Discover VS Constant Layout - -This is the **most critical** step. You must determine which VS constant registers hold View, Projection, and World matrices. - -**Static approach:** Decompile functions that call `SetVertexShaderConstantF`: -```bash -python -m retools.decompiler --types patches//kb.h -``` - -**Dynamic approach:** Trace `SetVertexShaderConstantF` calls live: -```bash -python -m livetools trace --count 50 \ - --read "[esp+8]:4:uint32; [esp+10]:4:uint32; *[esp+c]:64:float32" -``` -This captures: startRegister, Vector4fCount, and the actual float data (first 4 vec4 constants, dereferenced from `pConstantData`). - -**How to identify matrices:** -- View matrix: changes with camera movement, contains camera orientation -- Projection matrix: contains aspect ratio and FOV, rarely changes -- World matrix: changes per object, contains position/rotation/scale -- Look for 4x4 matrices (16 floats = 4 registers). Row 3 often has `[0, 0, 0, 1]` for affine transforms. - -### Step 3: Set Up Per-Game Project - -Copy the entire `rtx_remix_tools/dx/remix-comp-proxy/` folder to `patches//` (excluding `build/`). The game folder is now self-contained. Edit files directly: - -1. Edit register layout defaults in `src/shared/common/ffp_state.hpp` -2. Edit `src/comp/main.cpp`: set `WINDOW_CLASS_NAME` to the game's window class -3. Customize `src/comp/modules/renderer.cpp` draw routing if needed -4. Customize `src/comp/game/game.cpp` with game-specific hooks -5. Update `kb.h` with discovered function signatures, structs, and globals - -### Step 4: Build and Deploy - -```bash -cd patches/ -build.bat release --name -``` - -Deploy to game directory: `d3d9.dll` + `remix-comp-proxy.ini`. If using Remix, also place `d3d9_remix.dll` there. - -### Step 5: Diagnose with Log and ImGui - -The proxy writes `rtx_comp/diagnostics.log` in the game directory. After a configurable delay (default 50 seconds via `[Diagnostics] DelayMs`), it logs frames of detailed draw call data: - -- **VS regs written**: shows which constant registers the game actually fills -- **Vertex declarations**: what vertex elements each draw uses (POSITION, NORMAL, TEXCOORD, BLENDWEIGHT, etc.) -- **Draw calls**: primitive type, vertex count, index count, textures bound per stage -- **Matrices**: actual View/Proj/World values being applied - -Press **F4** to open the ImGui debug overlay, which shows the FFP debug tab with live draw call stats and state information. - -Use this to iterate: wrong matrices -> re-check register mapping. Missing textures -> adjust AlbedoStage. Objects at wrong positions -> world matrix register is wrong. - -## Architecture Details for Editing - -### Code Map: Edit vs Do-Not-Touch - -**Only edit sections marked YES or MAYBE:** - -| File / Section | Edit Per-Game? | -|----------------|----------------| -| `ffp_state.hpp` register layout defaults | **YES** -- set register layout | -| `remix-comp-proxy.ini` `[Skinning] Enabled=` | **YES** -- only after rigid FFP works | -| `remix-comp-proxy.ini` `[FFP] AlbedoStage=` | **YES** -- set albedo texture stage | -| `renderer.cpp` `on_draw_indexed_prim()` | **YES** -- main draw routing | -| `renderer.cpp` `on_draw_primitive()` | **YES** -- draw routing for non-indexed draws | -| `ffp_state.cpp` `setup_lighting()`, `setup_texture_stages()`, `apply_transforms()` | MAYBE -- tweak if game needs different FFP state | -| `ffp_state.cpp` `on_set_vertex_declaration()` | MAYBE -- element parsing; add extra usages if needed | -| `ffp_state.cpp` `on_set_vs_const_f()` | MAYBE -- dirty tracking | -| `d3d9ex.cpp` hook implementations | NO -- infrastructure | -| `ffp_state.cpp` `engage()` / `disengage()` | NO -- enter/leave FFP mode | -| `skinning.cpp` | NO -- infrastructure (no per-game edits) | -| `diagnostics.cpp` | NO -- logging infrastructure | -| `imgui.cpp` | NO -- debug overlay | - -### DrawIndexedPrimitive Decision Tree - -This is the routing logic in `renderer.cpp` `on_draw_indexed_prim()`: - -``` -viewProjValid? -+-- NO -> shader passthrough (transforms not captured yet) -+-- YES - +-- curDeclIsSkinned? - | +-- YES + skinning module -> skinning::draw_skinned_dip() - | +-- YES + no skinning -> shader passthrough - +-- NOT skinned - +-- !curDeclHasNormal -> shader passthrough (HUD/UI) - +-- hasNormal -> ffp_state::engage + rigid FFP draw -``` - -**Common per-game changes to this tree:** -- Game's world geometry omits NORMAL -> remove or change the `!cur_decl_has_normal()` filter -- Game has special passes (shadow, reflection) -> filter by shader pointer, render target, or vertex count -- Game draws UI with DrawIndexedPrimitive + NORMAL -> add a filter (e.g. check stride or texture) - -### DrawPrimitive Decision Tree - -``` -viewProjValid AND lastDecl AND !curDeclHasPosT AND !curDeclIsSkinned? -+-- YES -> ffp_state::engage (world-space particles, non-indexed geometry) -+-- NO -> shader passthrough (screen-space UI, POSITIONT, no decl, skinned) -``` - -### Skinning Data Flow - -When skinning is enabled via `[Skinning] Enabled=1` in `remix-comp-proxy.ini`: - -1. **`ffp_state::on_set_vertex_declaration()`** -- Parses `D3DVERTEXELEMENT9` array. If both BLENDWEIGHT and BLENDINDICES are present, sets `cur_decl_is_skinned_` and captures per-element byte offsets and types. - -2. **`ffp_state::on_set_vs_const_f()`** -- When a write hits registers >= `BoneThreshold` with count >= `BoneMinRegs` and divisible by `RegsPerBone`, stores `bone_start_reg_` and `num_bones_`. - -3. **`skinning::draw_skinned_dip()`** -- Locks the game's source vertex buffer, calls `expand_skin_vertex()` per vertex, caches results by hash key. - -4. **`skinning::upload_bones()`** -- Reads bone matrices from VS constants, transposes, uploads via `SetTransform(WORLDMATRIX(i))`. Sets `D3DRS_VERTEXBLEND` and `D3DRS_INDEXEDVERTEXBLENDENABLE`. - -5. **Draw** -- The expanded VB + shared declaration are bound, draw executes with FFP indexed vertex blending. After the draw, original VB/decl/textures are restored. - -### Key Component Notes - -- **`ffp_state::engage()` / `disengage()`**: `engage()` NULLs shaders, applies transforms, sets up texture stages. `disengage()` restores the game's shaders. Avoids redundant state changes between consecutive FFP draw calls. -- **`ffp_state::apply_transforms()`**: Reads from the VS constant array using the INI register settings and calls `SetTransform` with transposed matrices (D3D9 FFP expects row-major). -- **ImGui overlay (F4)**: Shows live draw call stats, FFP conversion counts, and shader pass-through counts for real-time debugging. - -## Analysis Scripts -- Entry Points, Not Endpoints - -The scripts below are fast first-pass scanners. They surface candidate addresses and call sites to give you a starting point. They do **not** replace deep analysis -- always follow up with `retools` and `livetools` to understand what is actually happening. - -| Script | What it surfaces | -|--------|------------------| -| `scripts/find_d3d_calls.py ` | D3D9/D3DX imports and call sites | -| `scripts/find_vs_constants.py ` | `SetVertexShaderConstantF` call sites and register/count args | -| `scripts/find_ps_constants.py ` | `SetPixelShaderConstantF/I/B` call sites and register/count args | -| `scripts/find_device_calls.py ` | Device vtable call patterns and device pointer refs | -| `scripts/find_render_states.py ` | SetRenderState args decoded by category (culling, blending, depth, fog) | -| `scripts/find_texture_ops.py ` | Texture pipeline: SetTexture stages, TSS ops, sampler states | -| `scripts/find_transforms.py ` | SetTransform/MultiplyTransform types (World, View, Projection, Texture) | -| `scripts/find_surface_formats.py ` | CreateTexture/RenderTarget/DepthStencil format extraction | -| `scripts/find_stateblocks.py ` | State block creation, recording, and apply patterns | -| `scripts/decode_fvf.py ` | FVF bitfield decode from SetFVF calls | -| `scripts/find_vtable_calls.py ` | D3DX constant table usage and D3D9 vtable calls | -| `scripts/decode_vtx_decls.py --scan` | Vertex declaration formats (BLENDWEIGHT/BLENDINDICES -> skinning) | -| `scripts/find_shader_bytecode.py ` | Embedded shader bytecode extraction (version, size) | -| `scripts/classify_draws.py ` | Draw call classification by state context (FFP/shader/hybrid) | -| `scripts/find_matrix_registers.py ` | Identify View/Proj/World registers (CTAB + frequency + layout suggestion) | -| `scripts/find_skinning.py ` | Consolidated skinning analysis: skinned decls, bone palettes, blend states, suggested INI | -| `scripts/find_blend_states.py ` | D3DRS_VERTEXBLEND + INDEXEDVERTEXBLENDENABLE + WORLDMATRIX transforms | -| `scripts/scan_d3d_region.py 0xSTART 0xEND` | Map all D3D9 vtable calls in a code region | - -Scripts are at `rtx_remix_tools/dx/scripts/`. - -## Common Pitfalls - -- **Concatenated WVP/VP instead of separate matrices**: This is the **#1 Remix porting mistake**. Remix requires separate World, View, and Projection matrices passed via `SetTransform`. If the game uploads a pre-multiplied WorldViewProj or ViewProj to a single register range, the proxy gets a combined matrix it can't decompose. **Fix**: find where the game multiplies W*V*P and hook to capture individual matrices *before* concatenation. Use `find_matrix_registers.py` to detect this. -- **Matrices look wrong**: D3D9 FFP `SetTransform` expects row-major matrices. The proxy transposes them. If the game stores matrices column-major in VS constants (the common case), the transpose is correct. If the game is already row-major, remove the transpose in `ffp_state::apply_transforms()`. -- **Everything is white/black**: The game's albedo texture might be on stage 1+ instead of stage 0. Set `AlbedoStage` in `remix-comp-proxy.ini` `[FFP]` section, or trace `SetTexture` calls to find the pattern. -- **Some objects render, others don't**: `on_draw_primitive()` routes by vertex declaration -- world-space draws (have decl, no POSITIONT, not skinned) engage FFP; screen-space/no-decl pass through. `on_draw_indexed_prim()` additionally filters out draws without NORMAL as likely HUD/UI. If world geometry is missing, check whether its vertex decl has NORMAL and whether `view_proj_valid()` is true when those draws happen. -- **Skinned meshes are invisible**: Enable skinning with `[Skinning] Enabled=1` in `remix-comp-proxy.ini`. Check the log for bone count and declaration issues. -- **Game crashes on startup**: The chain-loaded Remix DLL might not be present. Set `Enabled=0` in `remix-comp-proxy.ini` `[Remix]` section to test without Remix first. -- **Geometry at origin / piled up**: World matrix register mapping is wrong. Every object gets identity world transform. Re-examine VS constant writes. -- **Characters' world geometry shifts after a skinned draw**: After uploading bone matrices, WORLDMATRIX(0) is clobbered by bone[0]. The proxy sets world dirty so `apply_transforms()` re-applies the world matrix on the next rigid draw. If this still causes issues, the bone threshold register may overlap with the world matrix register range. - -## Notes -- Do not change the diagnostic logging delay (unless specified by the user). The delay is important to ensure the user is able to get into the game with actual geometry being drawn before the logs start, otherwise they may get lost in the initial burst of draw calls during loading. -- Tell the user when you want to launch a game and have them interact with it for logging or hooking purposes. They MUST interact with the game to have this task be useful. diff --git a/.cursor/rules/no-copium.mdc b/.cursor/rules/no-copium.mdc deleted file mode 100644 index a6db1600..00000000 --- a/.cursor/rules/no-copium.mdc +++ /dev/null @@ -1,39 +0,0 @@ ---- -description: Engineering standards -- no workarounds, no duct tape, no copium -alwaysApply: true ---- - -# No Copium - -## Principle - -Every change should make the codebase better, not just make the problem go away. If a solution needs a paragraph to justify why it's not a hack, it's a hack. - -## Remove - -- **Fixes in the wrong layer**: a guard on a canvas to suppress commits that a model should own. Put the fix where the problem originates. -- **Tolerance inflation**: widening deltas or adding retries to hide flaky behavior. If the value is wrong, find out why. -- **Catch-all exception swallowing**: `try/except Exception: pass` to hide symptoms. -- **Excessive error/null handling**: adding too many error/None "if" checks. If the error is expected, handle it. If unexpected, raise it. -- **God methods**: 200+ line functions doing multiple things. Break into named steps. Focus on cognitive load. Design for fewer indentation levels. -- **Leaky abstractions**: implementation details leaking into layers/modules that should be agnostic of one another. - -## Design For - -- **Single responsibility**: one component, one job. If you need "and" to describe it, split it. -- **Ownership**: the component that creates the problem owns the fix. -- **Minimal public surface**: expose what consumers need, nothing more. - -## Commit to the New Code - -- **No legacy fallbacks**: if you replace a system, remove the old one. -- **No dead code**: commented-out blocks, unused imports, orphan functions "just in case". Version control is the safety net. -- **No multiple paths to the same result**: one way to do each thing. If two paths exist, one is wrong. -- **No half-migrations**: finish the job -- update every reference, remove old APIs. - -## Smell Tests - -- "It works if I add a sleep" -- broken data flow. -- "It works if I read from widget instead of storage" -- the two are out of sync. -- "It passes alone but fails with other tests" -- shared mutable state leaking. -- "I added a flag to skip this code path" -- why does that path run in the first place? diff --git a/.cursor/rules/project-workspace.mdc b/.cursor/rules/project-workspace.mdc deleted file mode 100644 index 9e38b27e..00000000 --- a/.cursor/rules/project-workspace.mdc +++ /dev/null @@ -1,44 +0,0 @@ ---- -description: Project workspace conventions — patches/ directory, backups, and knowledge base format -alwaysApply: true ---- - -# Project Workspace - -Use `patches//` (git-ignored) for all project-specific artifacts: -- Knowledge base files (`kb.h`) -- One-off analysis scripts -- ASI patch specs and builds -- Notes, logs, collected trace data - -Create the project subfolder on first use. - -# Backups - -Before modifying project files (proxy source, kb.h, proxy.ini, build scripts, ASI specs), create a timestamped backup in `patches//backups/`: - -``` -patches//backups/YYYY-MM-DD_HHMM_/ -``` - -Copy ALL files being modified into the backup folder. The description should be a short slug of what the update does (e.g. `added-world-matrix-regs`, `fixed-albedo-stage`, `enabled-skinning`). - -Create the backup BEFORE making changes so it captures the last known-good state. This applies to all development work — FFP proxy edits, ASI patch specs, build config changes, and any other project file modifications. - -# Knowledge Base - -When reverse engineering a binary, maintain a knowledge base file (`.h`) that accumulates discoveries. Store in `patches//kb.h`. - -**Format:** C types (no prefix), functions (`@` prefix), globals (`$` prefix): -```c -struct Foo { int x; float y; }; -@ 0x401000 void __cdecl ProcessInput(int key); -$ 0x7C5548 Object* g_mainObject -``` - -**When to update the KB:** -- When you identify a function's purpose, add `@ 0xADDR` with a descriptive name and signature -- When you reconstruct a struct (e.g., from `structrefs.py --aggregate`), add the struct definition -- When you identify a global variable via `datarefs.py`, add `$ 0xADDR` with its name and type -- When you identify magic constants, define an enum with named values -- When `rtti.py` reveals a class name, use it in struct/function names diff --git a/.cursor/rules/subagent-workflow.mdc b/.cursor/rules/subagent-workflow.mdc deleted file mode 100644 index 1ae2e646..00000000 --- a/.cursor/rules/subagent-workflow.mdc +++ /dev/null @@ -1,143 +0,0 @@ ---- -description: Subagent delegation rules — when to delegate static analysis vs run livetools directly, parallel work patterns -alwaysApply: true ---- - -# Subagent Workflow - -The main agent orchestrates and focuses on **live tools**, **dx9tracer capture**, **user interaction**, and **synthesis**. Heavy static analysis and web research are delegated to subagents so the user isn't blocked. - -## Pre-flight: Ensure Ghidra Backend - -Before first use of pyghidra, the `static-analyzer` subagent should check if Ghidra is available. Run `python verify_install.py` — if pyghidra/Ghidra shows WARN, run `python verify_install.py --setup` to auto-download JDK 21 + Ghidra 11.4.3 + pyghidra. This is a one-time setup (~600MB download). Skip if pyghidra already shows PASS. - -## Bootstrap First — New Binaries - -When analyzing a binary for the first time (no existing or sparsely populated `patches//kb.h`), **always bootstrap before other static analysis**: - -1. The `static-analyzer` subagent auto-pulls `signatures.db` if missing (pre-flight check). Spawn it to run `bootstrap.py --project ` — this seeds `patches//kb.h` with RTTI classes, CRT/library function IDs, compiler info, and propagated labels. **Bootstrap takes 2-5 minutes.** Tell the user it's running and do other work while it completes. The output goes to `patches//kb.h` — verify this file exists and has content after bootstrap returns. **Bootstrap speeds up all subsequent decompilation**: when `--types kb.h` is passed to the decompiler, it pre-analyzes every known function (`af` per KB entry) so cross-references resolve to named functions, callees get inlined signatures, and you avoid the expensive full-binary `aaa` analysis pass. -2. **In parallel**, spawn a second `static-analyzer` to run `pyghidra_backend.py analyze --project patches/`. This runs Ghidra's full analysis (PE loader, MSVC calling convention detection, type propagation, RTTI parsing) and saves a reusable project. **Takes 5-15 minutes.** Once complete, all subsequent decompilations via `--backend auto --project patches/` will use Ghidra's higher-quality output. -3. Any other static analysis subagents should run in parallel, but their decompilation output will be richer if bootstrap finishes first -4. After bootstrap, all subsequent `decompiler.py` calls **must** use `--types patches//kb.h` -5. After pyghidra analyze, all subsequent `decompiler.py` calls should also use `--project patches/` so `--backend auto` prefers Ghidra when available - -**How to detect "needs bootstrap":** Check if `patches//kb.h` exists AND has real content (function signatures `@`, globals `$`, or struct definitions beyond section headers). An empty or stub KB with only comment headers counts as sparse — bootstrap it. Quick check: `grep -cE '^[@$]|^struct |^enum ' patches//kb.h` — if the count is under 50, bootstrap. - -**How to detect "needs pyghidra analyze":** Check if `patches//ghidra/.gpr` exists. If not, spawn `pyghidra_backend.py analyze`. If kb.h also needs bootstrap, spawn both in parallel. - -## Delegation Rules - -| Task | Where | -|------|-------| -| Static analysis (`retools`: decompiler, disasm, xrefs, search, structrefs, callgraph, rtti, datarefs, dumpinfo, throwmap) | `static-analyzer` subagent | -| Web research (docs, API refs, format specs, SDK docs) | `web-researcher` subagent | -| Live tools (`livetools`: attach, trace, bp, memwatch, dipcnt, mem read/write) | Main agent — directly | -| dx9tracer trigger/capture | Main agent — directly | -| dx9tracer analyze (offline JSONL analysis) | `static-analyzer` subagent | -| Bootstrap new binary (`bootstrap.py`) | `static-analyzer` subagent -- takes 2-5 min | -| pyghidra analyze (first-time Ghidra analysis) | `static-analyzer` subagent -- takes 5-15 min | -| Decompiler with `--backend ghidra` (subsequent) | `static-analyzer` subagent -- fast (JVM ~3s + decompile <1s) | -| Bulk signature scan (`sigdb.py scan`) | `static-analyzer` subagent -- takes 1-3 min | -| Signature DB build (`sigdb.py build`) | `static-analyzer` subagent -- takes 1-5 min | -| Single function ID (`sigdb.py identify`, `fingerprint`) | Main agent -- fast (<5s) | -| Context assembly (`context.py assemble`) | Main agent -- fast (<5s) | -| Decompiler postprocess (`context.py postprocess`) | Main agent -- instant | -| Dataflow: constants + backward slice (`dataflow.py`) | Main agent -- fast (<5s) | -| File editing, patch specs, builds | Main agent — directly | -| KB updates from subagent findings | `static-analyzer` writes to `kb.h`; main agent may refine | - -## Subagent Output Files - -Subagents write detailed findings to `patches//findings.md` (appended, not overwritten). When a subagent returns, it states the file path — **read the file** for full details including decompilation output, address tables, and suggested livetools commands. The return message is just a summary. - -## Parallel Work - -When both static and dynamic analysis are needed: -1. Spawn `static-analyzer` **in background** for the static questions -2. **Immediately ask the user** if the game/process is running or ask them to launch it — don't wait for static results -3. While the subagent works, prepare livetools (attach, set up traces) or discuss the approach with the user -4. Synthesize findings when the subagent returns - -Multiple `static-analyzer` instances can run in parallel for independent questions (e.g., decompiling two unrelated functions, analyzing different modules). When a subagent returns findings with multiple leads (e.g., "5 candidate functions found"), spawn parallel subagents to chase independent leads simultaneously — don't serialize them or try to analyze them yourself. - -## Dual-Backend Deep Analysis - -For deep analysis tasks (finding subsystems, mapping call chains, understanding large code areas), spawn **two parallel static-analyzer agents using different decompiler backends**: - -1. **r2ghidra agent** — uses `--backend pdg` (with `--types kb.h`), writes to `patches//findings_r2.md` -2. **pyghidra agent** — uses `pyghidra_backend.py decompile` (requires Ghidra project), writes to `patches//findings.md` - -**Why both:** Each backend has different strengths. r2ghidra is better at `__thiscall` recovery on small functions and low-level D3D details. pyghidra resolves more library calls, finds larger function scopes, and propagates types better. Neither finds everything alone — merging both gives the most complete picture. - -**When to use dual-backend:** Complex exploratory tasks ("find the culling system", "map the rendering pipeline", "understand the network protocol"). Not needed for single-function decompilation — use `--backend auto` for that. - -**Synthesis:** When both agents return, the main agent reads both findings files and merges them into a unified analysis. Conflicting information is resolved by checking which backend's output is more complete for that specific function. - -## Main Agent Responsibilities During Analysis - -**Do not silently wait for subagents.** While static analysis runs: -- Ask the user to launch the game/process if live verification or patching will be needed -- Discuss the approach, explain what the subagent is looking for -- Prepare livetools commands based on what you already know -- If the task involves runtime patching (disabling culling, skipping checks, etc.), assume live tools WILL be needed and prompt the user early - -## Examples - -**"Disable culling in game.exe"** -1. Spawn `static-analyzer` #1 (r2ghidra): find `SetRenderState` calls with `D3DRS_CULLMODE`, string search for "cull", xrefs --indirect to find vtable call sites. Uses `--backend pdg --types kb.h`. Writes to `findings_r2.md`. -2. Spawn `static-analyzer` #2 (pyghidra): same search strategy but decompile with `pyghidra_backend.py decompile`. Writes to `findings.md`. -3. Immediately tell the user: "Please launch the game — I'll need to attach with livetools to patch culling at runtime once I find the addresses" -4. While waiting, run `dataflow.py --constants` on any known render functions to see what cull mode constants flow in (e.g., `eax = 0x2` = D3DCULL_CW) -5. When both return, merge findings and use `livetools` to verify and patch: `mem write` to NOP the cull-enable instruction or force `D3DRS_CULLMODE` to `D3DCULL_NONE` - -**"What does function 0x401000 do?"** -1. Spawn `static-analyzer`: decompile with `--types kb.h`, get callgraph --indirect, xrefs -2. Run `dataflow.py 0x401000 --constants` inline — see what constants flow through -3. Tell the user: "Static analysis is running. Want me to also trace this function live to see actual register values and call frequency?" -4. If yes, attach with `livetools trace 0x401000 --count 20 --read` - -**"Find who writes to address 0x7A0000"** -1. Spawn `static-analyzer`: `datarefs.py` for static references -2. Ask user: "Is the game running? I can also set a `livetools memwatch` to catch runtime writes that static analysis might miss" -3. Combine static xrefs with live write traces for complete picture - -**"Why does the game crash in d3d9.dll?"** -1. Spawn `static-analyzer`: `dumpinfo.py diagnose`, `throwmap.py match` -2. Tell the user: "Analyzing the crash dump. If you can reproduce the crash, launch the game and I'll attach to catch it live" - -**"Analyze game.exe for the first time"** -1. Spawn `static-analyzer` #1 in background: `bootstrap.py game.exe --project MyGame` -2. Spawn `static-analyzer` #2 in background: `pyghidra_backend.py analyze game.exe --project patches/MyGame` -3. Tell the user: "Bootstrapping the binary and running Ghidra analysis in parallel. Bootstrap ~3 min, Ghidra ~10 min." -4. While both run, use `sigdb.py fingerprint` (fast) to tell the user the compiler version -5. When bootstrap returns, read the report and summarize coverage to the user -6. When pyghidra returns, tell the user: "Ghidra analysis complete. Subsequent decompilations will use Ghidra's higher-quality output." -7. All subsequent decompilations use `--types patches/MyGame/kb.h --project patches/MyGame` - -## Anti-Patterns - -**The Cascade Trap.** The main agent runs "one quick xref" -> sees an interesting caller -> decompiles it -> follows another xref -> now it's doing a full static analysis session while the user waits. If you catch yourself about to run a second retools command, stop and delegate everything to a subagent. - -**Duplicating subagent work.** After spawning a static-analyzer, don't also grep/search for the same thing yourself. Trust the subagent. Use the wait time for livetools or user interaction. - -**Silent waiting.** Spawning a subagent and then producing no output until it returns. Always talk to the user or do livetools work while subagents run. - -## When NOT to Delegate - -- Allowlisted fast commands (see CLAUDE.md Delegation Rule): `sigdb identify`, `sigdb fingerprint`, `context assemble`, `context postprocess`, `readmem.py`, `asi_patcher.py build` -- Anything requiring a live attached process — always main agent -- Iterative debugging loops where each step depends on the last live result — main agent - -Everything else in `retools.*` goes to a `static-analyzer` subagent. No exceptions. - -## Cursor Subagent Setup - -Cursor supports parallel subagent dispatch via the `Task` tool. To make it work: - -1. **Select a specific model** in the model dropdown (e.g. `claude-sonnet-4`, `gpt-4o`). Do NOT use "Auto" or "Composer" — these break the Task tool binding. -2. The `static-analyzer` and `web-researcher` agents in `.cursor/agents/` are loaded automatically. The parent agent reads their `description` fields to decide when to delegate. -3. **Parallel dispatch**: Send multiple `Task` calls in a single message to run subagents simultaneously. Each gets its own context window. -4. Subagents inherit all tools from the parent (including MCP tools). There is no `tools` field in Cursor agents — use `readonly: true` to restrict write access for read-only agents like `web-researcher`. -5. **`is_background: true`** makes a subagent non-blocking (parent continues while subagent works). Works at level 1 only — nested subagents block synchronously. - -**If subagent dispatch is unavailable** (wrong model selected, CLI mode, or Task tool not bound): follow the delegation rules yourself — do not run multiple retools commands in sequence. Collect all static analysis questions and run them in a single comprehensive pass. The principle is the same — avoid the Cascade Trap where "one quick xref" turns into a full analysis session. diff --git a/.cursor/rules/tool-catalog.mdc b/.cursor/rules/tool-catalog.mdc deleted file mode 100644 index 650f0828..00000000 --- a/.cursor/rules/tool-catalog.mdc +++ /dev/null @@ -1,333 +0,0 @@ ---- -description: Catalog of all RE tools -- pick the right tool for the job -alwaysApply: true ---- - -# Tool Catalog - -**BEFORE FIRST USE**: Run `python verify_install.py` from the repo root. Do NOT proceed with any tool until every required check passes. If pyghidra/Ghidra shows as WARN, run `python verify_install.py --setup` to auto-download JDK 21 + Ghidra + pyghidra. Common failures: missing `git lfs pull` (LFS pointer stubs instead of binaries), missing `pip install -r requirements.txt`. - -All tools work on PE binaries (`.exe` and `.dll`). `$B` = path to binary, `$VA` = hex address, `$D` = path to minidump `.dmp` file. Check tools help command for more info on usage. -Always consult this catalog before making any move to take the best decision on what to use with best bang for your buck. -Run all tools from the repo root directory using `python -m ` syntax (e.g. `python -m retools.search`). Do NOT modify files inside `retools/`, `livetools/`, or `graphics/` unless working on the tools themselves. - -IMPORTANT: Collecting MORE INFORMATION per command run is encouraged over minor snippets of data/output that don't reveal the whole picture. - -## Decision Guide - -### Run Directly (main agent) - -These are fast (<5s) and allowed inline: - -- "What compiler built this?" → `python -m retools.sigdb fingerprint $B` -- "Is this a known library function?" → `python -m retools.sigdb identify $B $VA` -- "Get full context before reasoning about a function" → `python -m retools.context assemble $B $VA --project $P` -- "Clean up decompiler output with known names" → pipe through `python -m retools.context postprocess` -- "Read a typed value from the PE file" → `python -m retools.readmem $B $VA $TYPE` -- "What constant flows into this register?" → `python -m retools.dataflow $B $VA --constants` -- "Trace where this value comes from" → `python -m retools.dataflow $B $VA --slice TARGET_VA:REG` -- "Build an ASI patch DLL" → `python -m retools.asi_patcher build spec.json` -- "Does a Ghidra project exist for this binary?" → `python retools/pyghidra_backend.py status $B --project $P` - -### Delegate to `static-analyzer` subagent - -Everything else. Tell the subagent WHAT you need, not HOW to run it — it has the full tool catalog. - -**D3D9-specific questions?** Check the DX analysis scripts section below first — they're faster and more targeted than general retools for D3D API usage, device calls, shader constants, and vertex formats. - -- "What does this function do?" → decompile + callgraph + xrefs + dataflow --constants -- "Who calls this function?" → xrefs or callgraph --up -- "What does this function call?" → callgraph --down (add --indirect for vtable calls) -- "Who calls this virtual method?" → xrefs --indirect + filter by vtable slot offset -- "What constant reaches this call?" → dataflow --constants or --slice VA:REG -- "Resolve a switch/jump table" → cfg (auto-resolves MSVC switch patterns) -- "Find a string and who uses it" → string search with xrefs -- "Where is this global read/written?" → datarefs -- "Where is struct field +0x54 used?" → structrefs -- "What does this struct look like?" → structrefs --aggregate -- "What C++ class is this vtable?" → RTTI resolution -- "What type was a caught/thrown exception?" → RTTI throwinfo -- "Find instructions using a specific constant" → instruction search -- "What crashed and what was the error message?" → dump diagnosis + throwmap -- "Map all throw sites to error strings" → throwmap list -- "First time analyzing a binary?" → bootstrap (2-5 min) + pyghidra analyze (5-15 min) in parallel -- "Bulk signature scan" → sigdb scan (1-3 min) -- Any combination of the above - -### Live tools (main agent, requires attached process) - -- "Is this function reached at runtime?" → `livetools trace` or `collect` -- "What are the actual register values?" → `livetools trace --read` or `bp` + `regs` -- "How many draw calls happen?" → `livetools dipcnt` -- "Who writes to this memory address?" → `livetools memwatch` -- "Send keys/clicks to the game window?" → `livetools gamectl` - -### DX analysis scripts (main agent, fast first-pass) - -These are targeted D3D9 scanners under `rtx_remix_tools/dx/scripts/`. They run in seconds and surface D3D-specific patterns that general-purpose retools would take longer to find. **Use these BEFORE retools** when the question is about D3D9 API usage, device calls, shaders, or vertex formats. Run as `python rtx_remix_tools/dx/scripts/