Expose read-only version datasets to every mod API level

.github/workflows/api3-contract.yml

@@ -0,0 +1,61 @@
+name: API 3 contracts
+on:
+  pull_request:
+    paths:
+      - '.github/workflows/api3-contract.yml'
+      - 'src/**'
+      - 'tests/api3_contract/**'
+      - 'tests/engine/api3_*.lua'
+      - 'tools/api3-contract-check/**'
+      - 'tools/flag-catalog-check/**'
+  push:
+    branches: [main, dev]
+    paths:
+      - '.github/workflows/api3-contract.yml'
+      - 'src/**'
+      - 'tests/api3_contract/**'
+      - 'tests/engine/api3_*.lua'
+      - 'tools/api3-contract-check/**'
+      - 'tools/flag-catalog-check/**'
+permissions:
+  contents: read
+concurrency:
+  group: api3-contract-${{ github.ref }}
+  cancel-in-progress: true
+jobs:
+  compare:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          path: gen2
+          sparse-checkout: |
+            src
+            data
+            tests
+            tools/api3-contract-check
+            tools/flag-catalog-check
+      - uses: actions/checkout@v7
+        with:
+          repository: bryanthaboi/gen1recomp
+          ref: 79ea62f45fb2ea8f65d081a109431e4eb1bbdb36
+          path: gen1-reference
+          sparse-checkout: |
+            src
+            data
+            tests
+            assets/fonts
+      - name: install LuaJIT
+        run: sudo apt-get update && sudo apt-get install -y luajit
+      - name: native flag catalog provenance
+        run: python3 gen2/tools/flag-catalog-check/generate.py --gen1 gen1-reference
+      - name: identical mod contracts on both engines
+        run: >-
+          python3 gen2/tools/api3-contract-check/check.py
+          --gen1 gen1-reference --gen2 gen2
+          --report "${{ runner.temp }}/api3-report.json"
+      - uses: actions/upload-artifact@v7
+        with:
+          name: api3-contract-report
+          path: ${{ runner.temp }}/api3-report.json
+          if-no-files-found: error

docs/api3-flags.md

@@ -0,0 +1,75 @@
+# API-3 native and private flags
+
+`mod.world:getFlag(reference)` returns a boolean or `nil, reason`.
+`setFlag(reference, boolean)` returns `true` or `nil, reason`. Real booleans are
+required. The facade follows the current Game/save without bootstrapping one.
+Flag reads/writes require a save but can run during menus/scripts; they do not
+start an idle-world action. Writes from a rolled-back/reloaded/unloaded owner
+are retired. Gen4/5 remain outside the shared API-3 world surface.
+
+## Private flags
+
+`mod:<key>` and `mod.<this-mod-id>.<key>` address the same private flag in
+`save.modData[modId].worldFlags`. A foreign `mod.<other-id>.*` reference is
+refused. Keys are nonempty strings up to 256 bytes without control characters.
+A read does not create storage. A clear removes the key. Malformed stores are
+refused without overwriting them. The existing change-only `flag.changed`
+payload remains `{name="mod.<modId>.<key>", value=boolean}`.
+
+## Cartridge flags
+
+| Version | Public references | Host storage |
+| --- | --- | --- |
+| Red/Blue | Known event symbol or its named bit index | Existing `EVENT_*` save.flags key |
+| Yellow | Yellow event symbol or its named bit index | Existing `EVENT_*` save.flags key, with Yellow's own bit catalog |
+| Gold/Silver | EVENT symbol or whole bit index 0–2047 | `EVENT_G2_%04d`, as used by the VM/save converter |
+| Crystal | Whole bit index 0–2047 | Same native numbered namespace; symbolic references explicitly decline, matching the pinned Gen1 API |
+| FireRed/Emerald | Known native flag symbol/alias, or a named numeric flag in that version's range | `FLAG_G3_%04X`, as used by the VM/save converter |
+
+Native IDs/symbols are cartridge-specific; no equivalence between the same bit
+number in different games is invented. Unknown, fractional, non-finite,
+negative, malformed or out-of-range references decline before writes. Special
+variables are not a new flag namespace. Prism/Polished Crystal native mappings
+are not guessed from Gold; private flags remain available there.
+
+Clearing a native host flag retains `false`, matching native object visibility's
+distinction between "cleared" and "untouched". Gen3 synchronizes active object
+templates carrying that exact native flag through the host's targeted native
+visibility entry; unrelated actors/templates are left alone. Native failures
+there can happen after the flag write and are not transactional rollback.
+
+Native `flag.changed` fires only on a true boolean transition. Its name is the
+Gen1 symbol, Gen2 bit number, or Gen3 version-catalog name (falling back to the
+numeric ID); Gen3 also provides `id`. For Gen3 cross-engine listeners, use
+`event.id` for native identity: the pinned Gen1 implementation's native emitter
+uses its base FireRed name catalog even on Emerald, so `event.name` is not a
+portable label there. The shared fixture consumes `event.id or event.name` on
+both engines. This preserves meaningful native identity rather than claiming
+all label strings are identical. Reentrant writes from an API-3 flag event or
+native visibility callback decline with "world is busy"; reads remain usable.
+
+## Catalog provenance and verification
+
+Generated Red/Yellow/FireRed/Emerald identifier catalogs contain names/numbers
+only. No Gen1 engine implementation or ROM is vendored. Gold/Silver reuse the
+host's existing Gen2Flags table. The reference is pinned to clean
+`bryanthaboi/gen1recomp@79ea62f45fb2ea8f65d081a109431e4eb1bbdb36`.
+
+```sh
+luajit tests/engine/api3_flags.lua
+python3 tools/flag-catalog-check/generate.py --gen1 /clean/pinned/gen1 --lua luajit
+# Add --write to regenerate the deterministic catalogs.
+```
+
+The provenance checker verifies every generated row and all 1235 pinned Gold
+symbol mappings against the host. CI checks provenance alongside the shared
+mod suite. Eight identical public-only mod cases bring that suite to
+72 comparisons / 144 isolated loader runs; compare preflight, numeric/symbolic
+aliases, booleans, no-op event suppression, public event identity/value and
+private ownership. The reference uses actual flag backends with synthetic
+world/store setup.
+
+Local checks cover every shipped catalog alias/named numeric entry, the Gen3
+VM key conversion and all 2048 Gen2 bits, plus malformed inputs/stores,
+owner retirement, reentry and scoped visibility dispatch. These are ROM-free
+runtime/store tests, not cartridge story progression or visual gameplay tests.

docs/api3-invalidate-map.md

@@ -0,0 +1,57 @@
+# API-3 map invalidation
+
+`mod.world:invalidateMap(mapId)` requests the host's native cache invalidation
+and reload for generations 1–3. Capture `mod.world` at entry and call later from
+an idle world. Return `true` or `nil, reason`; query availability through
+`mod:supports("world", "invalidateMap")`. Gen4/5 decline. API-1/2 behavior and
+the portable script vocabulary are unchanged.
+
+```lua
+local mod = ...
+local world = mod.world
+local function refresh(mapId)
+  return world:invalidateMap(mapId)
+end
+```
+
+A known map ID is required: a nonempty string up to 256 bytes without control
+characters. Invalid or unknown IDs return `nil, "unknown map"`, matching Gen1's
+API-3 preflight. Gen3 also resolves unprefixed names using the current version's
+FR_/EM_/RU_/SA_ prefix when no exact ID exists. Map IDs remain dataset-specific.
+
+This shares NPC/object/block mutation admission and reentry protection. A save
+and topmost idle overworld are required. Scripts, queued/parallel work, player
+movement, transitions, menus, link sessions and controller locks refuse before
+cache writes. Missing native reload support declines. Reading the facade never
+constructs Game. Rollback/reload/teardown retire retained mutation facades.
+
+The host's native reloadMap invalidates the target and resident neighbor cache
+entries. For the active map it uses setMap with seamless reload, current player
+cell/facing and keepMusic. If the retained player cell no longer fits the new
+geometry, native reload sends the player to the heal point. Runtime map/actor
+instance identity can change, so obtain fresh NPC handles after a reload.
+Inactive invalidation does not move the player or force an unrelated active
+map reload. Rendering and map lifecycle events remain native, not a new common
+renderer/cache implementation. Successful host invalidation emits map.reloaded
+with mapId and reason="invalidate". Pinned Gen1's Gen2 backend does not emit
+that event for inactive invalidation; this native difference is retained.
+
+This is a cache/instance operation, not a content-write API. Permanent map
+changes belong in content definitions. A native failure returns nil with its
+reason but may occur after cache invalidation; already entered native effects
+are not transactionally rolled back.
+
+## Verification
+
+Run `luajit tests/engine/api3_invalidate.lua` and the pinned checker documented
+in [API-3 portability](api3-portability.md). Eight identical public mod cases
+extend the full checker to 64 comparisons / 128 isolated loader runs. Real
+loaders/native facades compare preflight, active reload, position retention,
+notification and inactive-map acceptance; reload timing is a driver seam.
+
+Host tests additionally run actual OverworldState.reloadMap and
+MapLoader.invalidate against a synthetic cache, with the setMap/render entry
+instrumented. Verify target/neighbor eviction, seamless/music options, valid
+player retention, heal-point fallback, reentry, failure/guard release, busy
+worlds and retired owners. No ROM import or rendered gameplay equivalence is
+claimed.

docs/api3-npc.md

@@ -0,0 +1,142 @@
+# API-3 public NPC actions
+
+Generations 1–3 support the same public creation, lookup and removal calls as
+Gen1Recomp. Check availability and retain the world facade at mod entry:
+
+```lua
+local mod = ...
+local world = mod.world
+
+local function greet(mapId, spriteId)
+  if not mod:supports("world", "spawnNpc") then return end
+  local id, why = world:spawnNpc(mapId, {
+    name = "GREETER", sprite = spriteId, x = 3, y = 4, facing = "down",
+  })
+  if not id then return nil, why end
+  local npc, reason = world:npc(mapId, id)
+  if not npc then return nil, reason end
+  return npc:scriptMove("down", 2, function()
+    npc:face("left")
+  end)
+end
+```
+
+Invoke this later from an idle world. Reading the facade never bootstraps Game.
+Gen4/5 report these three calls unavailable. API-1/2 retain their native behavior;
+this does not add portable `queueScript` verbs.
+
+## Creation, lookup and ownership
+
+- `spawnNpc(mapId, definition)` returns the native runtime ID or `nil, reason`.
+  It can add a runtime definition to a known inactive map; an actor appears when
+  the native map lifecycle instantiates it. The return value promises a recorded
+  definition, not immediate live lookup or a common lifecycle event order.
+- `npc(mapId, reference)` returns an opaque handle or `nil, reason`. References
+  are the live actor's string ID, its definition's numeric index or its name.
+  Only the active map and currently live neighboring map actors can be queried.
+- `removeNpc(id)` returns `true` or `nil, reason`. Only runtime objects owned by
+  the calling mod may be removed. Imported objects and other owners are refused.
+  A mod may still look up and control an imported live actor.
+
+Map, sprite and object identifiers remain dataset-specific. Gen3 recognizes the
+current version's `FR_`, `EM_`, `RU_` or `SA_` map prefix when an exact map ID is
+absent. IDs/names are nonempty strings of at most 256 bytes without control
+characters; numeric references are finite nonnegative integers. Use the ID
+returned by `spawnNpc` for removal.
+
+Definitions are copied before native construction. Required `x` and `y` are
+whole nonnegative walk-cell coordinates, at most 1,000,000. Optional `facing` is
+`up`, `down`, `left` or `right`; optional `name` and `sprite` are identifiers.
+A supplied sprite must exist in the active sprite registry. Native generation
+fields remain supported as data; this API does not invent a canonical sprite,
+movement or NPC definition schema. The host assigns runtime ownership and
+identity, ignoring caller-supplied ownership/index markers.
+
+The copy accepts plain acyclic tables, strings, finite numbers and booleans.
+Functions, userdata, metatables and cycles are refused before native writes.
+Limits are depth 16, 4096 visited values/keys, 256 entries per table, 4096 bytes
+per string and 64 KiB total string bytes. Refusal returns `nil, reason`.
+
+## Handles
+
+Handles expose `id` and these methods; they never expose the native actor:
+
+| Method | Result and behavior |
+| --- | --- |
+| `position()` | `x, y` in zero-based 16-pixel walk cells, in the actor's own map frame. |
+| `isMoving()` | Boolean native movement state. |
+| `canStep(direction)` | Boolean native collision permission, including the host's current seam cast for connected maps. It is a query, not a reservation. |
+| `face(direction)` | Set facing. |
+| `stepNow(direction)` | Begin one authored step; refuse an already moving actor. |
+| `placeAt(x, y, facing?)` | Place at whole nonnegative coordinates up to 1,000,000, clear in-flight movement and synchronize pixel position. |
+| `setPassable(boolean)` | Set the live actor's passability. |
+| `setAppearance(spriteId)` | Refresh a live actor using an existing sprite registry entry. |
+| `scriptMove(direction, tiles?, onDone?)` | Queue native movement, default one cell; `tiles` must be an integer from 1 to 1024. |
+| `marchInPlace(onDone?)` | Queue one native stationary march. |
+
+Directions are the four lowercase names above. Mutations return `true` or
+`nil, reason`; a stale handle returns `nil, "NPC handle is stale"`. Handles are
+bound to the original actor, map instance, Game, save, data and version. Reused
+native IDs or editing the public `id` field cannot redirect a retained handle.
+The facade follows the current Game, while old actor handles do not.
+
+Spawn, removal and handle mutations use the shared API-3 idle-world guard:
+a save and topmost overworld are required; menus, native/portable/parallel
+scripts, player movement, controller locks, transitions and link sessions refuse
+admission. Reads remain usable during overlays. Synchronous reentry into NPC or
+object/block mutations is refused. Movement tracks require the active map;
+other handle calls also work for live neighboring actors.
+
+`stepNow` and `placeAt` are authored operations, matching the native distinction
+from `canStep`: they do not automatically perform collision validation or clamp
+to map bounds. Choose coordinates and query collision as appropriate for the
+scene. Appearance changes construct a renderer before committing the actor
+change, retain its cell position and leave the native template sprite intact;
+they are not persistent content edits.
+
+## Movement and mod lifetime
+
+Completion callbacks are optional, one-shot and attributed to the initiating
+mod. Native synchronous completion is deferred until admission finishes and
+the shared mutation guard is released. Native callback arguments, including
+nil holes, are preserved. Callback errors are contained and reported once.
+
+Pending completions are permanently retired on observed Game/save/data/world,
+map, stack, version, actor or runtime-bus replacement, and on native map lifecycle
+events. The input-step boundary observes replacements before and after native
+updates. Retiring removes only the movement entries owned by that operation;
+it does not pop screens or cancel another mod's tracks.
+
+Rollback, reload and teardown retire the mod's facade and pending tracks and
+remove its recorded runtime definitions/actors, including definitions spawned
+on inactive maps and instantiated later as neighbors. Foreign runtime objects
+and imported actors remain. Controls already applied to imported actors are
+ordinary native scene changes, not a transactional patch with automatic undo.
+Retained handles/facades return `nil, "world facade is retired"` after retirement.
+
+## Verification
+
+```sh
+luajit tests/engine/api3_npc.lua
+python3 tools/api3-contract-check/check.py \
+  --gen1 /path/to/clean/pinned/gen1recomp --lua luajit \
+  --report /tmp/api3-contract-report.json
+```
+
+The clean Gen1 reference stays pinned to
+`79ea62f45fb2ea8f65d081a109431e4eb1bbdb36`. The same public-only NPC mod runs
+through both production loaders for Red, Blue, Yellow, Gold, Silver, Crystal,
+FireRed and Emerald. Its eight cases extend the full checker to 48 comparisons
+and 96 isolated runs. The shared fixture compares creation/lookup, detached
+definitions, collision/passability, appearance, authored/queued movement,
+callback replay/arguments, optional callbacks and ownership-aware removal.
+
+Local tests additionally cover bounded validation, busy worlds, reentry, stale
+sessions/handles, callback errors, lifecycle cancellation and scoped cleanup.
+They exercise the host's actual NPC constructor, native runtime add/remove,
+scriptMove/marchInPlace/updateScriptMoves, NPC updates and seam collision casts.
+Synthetic datasets and a deterministic SpriteRenderer construction seam keep
+these ROM-free. The tests verify the documented shared subset, not retail-ROM
+playthroughs, GPU artwork or identical native internals/event timing. Host
+validation and lifecycle guards are intentionally stricter than some reference
+native paths.