SLM-267: streaming uniform-policy ProgramSpec corpus CLI + saturation reading

docs/design/compiler-inverted-program-data.md

@@ -0,0 +1,163 @@
+# Compiler-inverted ProgramSpec corpus (SLM-267 / VSD2-02)
+
+Status: **wiring-first increment; uniform-policy CLI landed; VSD-H7a
+inconclusive at smoke scale with a real throughput blocker identified.**
+
+This is the first PR against
+[SLM-267 (VSD2-02)](https://linear.app/quickdeploy-ai/issue/SLM-267/vsd2-02-scale-compiler-inverted-programspec-data-to-10k100k1m-with):
+scale the repository's typed `ProgramSpec`-first generator
+(`src/slm_training/data/progspec/generate.py`) into a streaming,
+coverage-audited semantic data engine and publish nested 10k/100k/1M corpora
+under uniform and coverage-targeted sampling. That full scope is far larger
+than one increment; this PR lands the streaming/resumable **uniform-valid**
+control CLI the issue specifies and uses it to take an honest, measured first
+reading of **VSD-H7a** ("the bounded OpenUI ProgramSpec space can supply
+nested 10k/100k/1M corpora"). Coverage-targeted sampling (needs a
+`CoverageGapManifestV1` consumer wired to SLM-265's outcome-blind domain-shift
+strata), the 100k/1M rungs, and the in-issue training experiment are explicit
+follow-on scope — not attempted here.
+
+## What shipped
+
+`python -m scripts.generate_programspec_corpus` (new;
+`tests/test_scripts/test_generate_programspec_corpus.py`):
+
+- Wraps the existing `ProgramGenerator` — never replaces it.
+- `--policy uniform` only in this increment (`--policy targeted` is not a
+  valid choice yet; the issue's `--policy` surface is preserved so a follow-up
+  can add it without a CLI break).
+- Deterministic `(seed, shard_id)` generation over a `GeneratorConfig`
+  (`--max-depth`, `--max-width`, `--components`).
+- Disk-backed **exact** dedup: the accepted set is keyed on
+  `ProgramSpec.id` (already a canonical content hash of the program +
+  viewport/depth/width/prop-target identity computed inside the generator) —
+  no Bloom-filter-only path exists.
+- Immutable rejection ledger (`rejected.jsonl`): every `generate_one()`
+  `ValueError` other than grid exhaustion is appended with its call index and
+  verifier-failure reason, never silently dropped.
+- Resumable `state.json` (schema `programspec_corpus_stream_state/v1`) and a
+  content-addressed `manifest.json` (schema
+  `programspec_corpus_stream_manifest/v1`, `accepted_ids_sha256` over the
+  sorted accepted-id set) under the canonical `DataStore` `programspec` root
+  (`outputs/data/programspec/<dataset_id>/`) — no shadow artifact tree.
+- `--describe` dry-run; `--max-wall-minutes` capped at
+  `slm_training.levers.MAX_RUN_MINUTES` like every other capped harness in
+  this repo.
+
+## Determinism contract (why resume is possible at all)
+
+For a fixed `GeneratorConfig` + `seed`, `ProgramGenerator` visits candidates in
+an identical order and produces an identical accept/reject sequence on every
+process invocation — nothing in candidate selection or verification consumes
+wall-clock time, OS randomness, or network state. `tests/test_scripts/test_generate_programspec_corpus.py::test_resume_produces_monotonic_prefix`
+locks this: resuming with a larger `--target-unique-roots` always reproduces
+the prior accepted-id prefix byte-for-byte before extending it.
+
+## Measured evidence (smoke scale, local CPU, two capped shards)
+
+Command (both shards identical except invocation time):
+
+```bash
+python -m scripts.generate_programspec_corpus \
+  --pack openui --policy uniform --target-unique-roots 10000 \
+  --seed 0 --shard-id 0 --max-wall-minutes 2.5 \
+  --dataset-id compiler_programs_uniform_smoke_v1
+```
+
+Config: default (all 54 pinned components), `max_depth=3`, `max_width=3` —
+the same defaults `ProgramGenerator` and `GeneratorConfig` ship with.
+
+| Shard | Calls this run | New accepted | New rejected | Cumulative accepted | Disposition |
+| --- | ---: | ---: | ---: | ---: | --- |
+| 1 (fresh) | 334 | 334 | 0 | 334 | `wall_clock_stopped` |
+| 2 (resume) | 324 | **0** | 0 | 334 | `wall_clock_stopped` |
+
+Full evidence:
+[`iter-slm267-uniform-saturation-20260725.json`](iter-slm267-uniform-saturation-20260725.json).
+
+Zero verifier rejections at these settings (all 334 generated programs cleared
+G0–G10 as Silver) — the earlier 0/50-accept reading during local development
+was an environment artifact (the `@openuidev/lang-core` Node bridge under
+`src/apps/openui_bridge/` was not `npm ci`-installed), not a generator defect;
+confirmed by the full `tests/test_data/test_progspec_generate.py` suite
+passing once the bridge was installed.
+
+## VSD-H7a reading: `generator_state_space_saturates` risk confirmed, not yet resolved
+
+Two independent, measured findings bear directly on the issue's falsifier
+("the state space, prompt inversion, verifier, or dedup contract is
+materially narrower than assumed"):
+
+1. **The candidate grid is a small, seed-independent, near-constant function
+   of `GeneratorConfig`, not of the target root count.** Measured
+   `len(ProgramGenerator(config)._candidates)`:
+
+   | `max_depth` | `max_width` | Candidates (54 components) |
+   | ---: | ---: | ---: |
+   | 3 | 3 | 1,781 |
+   | 5 | 4 | 1,785 |
+   | 8 | 6 | 1,791 |
+   | 10 | 8 | 1,793 |
+
+   Depth/width barely move the count because the grid is dominated by
+   `O(components²)` pairwise-coverage candidates and
+   `O(components × properties × variants)` prop-target candidates, not by
+   depth/width combinatorics. Critically, **`seed` only perturbs tie-break
+   jitter among equal-coverage-score candidates — it never adds new
+   candidates.** Sharding by `(global_seed, shard_id, worker_id)` as the issue
+   specifies therefore cannot by itself grow reachable diversity past 1,781
+   unique roots for this config; only widening `GeneratorConfig` (more
+   `selected_groups`/triples, literal-value pools, multi-instance repeats
+   within one program) can. That widening is unscoped in this PR.
+
+2. **The MVP resume mechanism does not deliver practical cross-shard
+   throughput at measured verifier cost.** At ≈0.45–0.46s/`generate_one()`
+   call (bridge-subprocess + full G0–G10 verification per candidate), a
+   from-scratch shard nets ~330 new unique roots in one capped 150s window.
+   Because this increment's resume strategy re-derives state by **replaying**
+   the deterministic call sequence from index 0 (skipping emission, not
+   verification, for already-seen calls) rather than persisting the
+   generator's internal candidate cursor, the *second* capped shard spent its
+   entire budget re-executing the first shard's 334 calls and landed **zero**
+   net-new records. Total work to reach cumulative count `M` across many
+   small shards is closer to `O(M·shards)` than `O(M)` — the opposite of
+   "resumable" at this cost level.
+
+Neither finding is a final verdict on VSD-H7a: the smoke run used the
+smallest, un-widened `GeneratorConfig` and the naive resume path. Both are
+concrete, falsifiable blockers a follow-up increment must clear before a 10k
+uniform corpus is reachable:
+
+- persist the generator's internal `_used` candidate cursor (or an equivalent
+  index) in `state.json` so resume is `O(new work)`, not `O(total work)`;
+- widen `GeneratorConfig`/candidate construction (or add a second axis of
+  variation) so the candidate grid itself can exceed 10k before the resume
+  question even matters.
+
+## Disposition
+
+**`inconclusive`** (one of the issue's own listed acceptance dispositions).
+VSD-H7a is neither confirmed nor falsified: this increment did not reach the
+10k rung, but it also did not exhaust the current 1,781-candidate grid (334 of
+1,781, `exhausted: false`) — it hit the resume-throughput blocker first. No
+training experiment, coverage-targeted policy, or 100k/1M rung was attempted.
+No checkpoint, ship, or promotion claim is made or implied.
+
+## Non-goals of this PR
+
+- No `--policy targeted` implementation (needs a `CoverageGapManifestV1`
+  consumer; SLM-265's `iter-slm265-domain-shift-audit-20260724.json` is
+  `coverage_gap_manifest/v1`-shaped evidence but is not yet exposed as a
+  stable manifest artifact for a generator to consume).
+- No `publish_programspec_corpus.py` / `DataStore.publish` — this stays a
+  local, inspectable staging corpus under `outputs/data/programspec/`, not a
+  durable Git-published dataset.
+- No 100k/1M rung, no 3-seed training-arm comparison, no checkpoint.
+
+## Related
+
+- Generator: [`src/slm_training/data/progspec/generate.py`](../../src/slm_training/data/progspec/generate.py)
+- New CLI: [`scripts/generate_programspec_corpus.py`](../../scripts/generate_programspec_corpus.py)
+- Tests: [`tests/test_scripts/test_generate_programspec_corpus.py`](../../tests/test_scripts/test_generate_programspec_corpus.py)
+- SLM-265 evidence consumed for status only (not as a coverage manifest):
+  [`iter-slm265-domain-shift-audit-20260724.json`](iter-slm265-domain-shift-audit-20260724.json)

docs/design/iter-slm267-uniform-saturation-20260725.json

@@ -0,0 +1,93 @@
+{
+  "schema": "slm267_uniform_saturation_reading/v1",
+  "issue": "SLM-267",
+  "hypothesis": "VSD-H7a",
+  "status": "inconclusive",
+  "objective": "First measured reading of whether the bounded OpenUI ProgramSpec space can reach 10k unique canonical roots under the frozen uniform-valid policy, using the new streaming/resumable CLI.",
+  "recipe": {
+    "command": "python -m scripts.generate_programspec_corpus --pack openui --policy uniform --target-unique-roots 10000 --seed 0 --shard-id 0 --max-wall-minutes 2.5 --dataset-id compiler_programs_uniform_smoke_v1",
+    "device": "cpu",
+    "generator_config": {
+      "components": "all_pinned (54)",
+      "max_depth": 3,
+      "max_width": 3
+    },
+    "shards_run": 2,
+    "max_wall_minutes_per_shard": 2.5,
+    "honesty_mode": "smoke_local_no_checkpoint_no_training"
+  },
+  "candidate_grid_size_by_config": [
+    {"max_depth": 3, "max_width": 3, "candidates": 1781},
+    {"max_depth": 5, "max_width": 4, "candidates": 1785},
+    {"max_depth": 8, "max_width": 6, "candidates": 1791},
+    {"max_depth": 10, "max_width": 8, "candidates": 1793}
+  ],
+  "seed_independence_note": "candidate count is a deterministic function of GeneratorConfig only; seed perturbs tie-break jitter among equal-score candidates and never adds candidates.",
+  "shard_reports": [
+    {
+      "shard_index": 1,
+      "fresh_start": true,
+      "calls_this_run": 334,
+      "new_accepted": 334,
+      "new_rejected": 0,
+      "cumulative_accepted": 334,
+      "cumulative_calls": 334,
+      "disposition": "wall_clock_stopped",
+      "elapsed_seconds": 150.0,
+      "per_record_seconds": 0.4491017964071856
+    },
+    {
+      "shard_index": 2,
+      "fresh_start": false,
+      "calls_this_run": 324,
+      "new_accepted": 0,
+      "new_rejected": 0,
+      "cumulative_accepted": 334,
+      "cumulative_calls": 658,
+      "disposition": "wall_clock_stopped",
+      "elapsed_seconds": 150.0,
+      "per_record_seconds": 0.46296296296296297
+    }
+  ],
+  "final_manifest": {
+    "schema": "programspec_corpus_stream_manifest/v1",
+    "dataset_id": "compiler_programs_uniform_smoke_v1",
+    "pack": "openui",
+    "policy": "uniform",
+    "seed": 0,
+    "shard_id": "0",
+    "target_unique_roots": 10000,
+    "unique_roots": 334,
+    "candidate_grid_size": 1781,
+    "rejected_total": 0,
+    "calls_total": 658,
+    "exhausted": false,
+    "accepted_ids_sha256": "1a7b75bcc58e2f9d9c42205c394ac6a91675288da347b56bbcddcee067eae719"
+  },
+  "findings": [
+    "All 334 generated programs cleared G0-G10 as Silver at this config; zero rejections at smoke scale.",
+    "The uniform-policy candidate grid (1,781 at the config used here) is far below the 10k target and is nearly insensitive to --max-depth/--max-width, because it is dominated by O(components^2) pairwise and O(components*properties*variants) prop-target candidates rather than depth/width combinatorics.",
+    "The second capped shard replayed the first shard's 324-of-334 prior calls and produced zero net-new accepted records: the MVP resume mechanism (replay from index 0, skip emission for already-seen calls) does not deliver O(new work) cross-shard throughput at the measured ~0.45-0.46s/generate_one() cost -- it is closer to O(total work) per shard.",
+    "The 0/50-accept reading seen during local development before this run was traced to a missing `npm ci` install for the @openuidev/lang-core Node bridge (src/apps/openui_bridge/), not a generator or verifier defect; the full tests/test_data/test_progspec_generate.py suite passes once the bridge is installed."
+  ],
+  "blockers_for_next_increment": [
+    "Persist the generator's internal candidate cursor (not just accepted ids) in state.json so resume is O(new work).",
+    "Widen GeneratorConfig / candidate construction (more selected_groups/triples, literal-value pools, multi-instance component repeats) so the candidate grid itself can exceed 10k before the resume-cost question is the binding constraint.",
+    "Wire a stable CoverageGapManifestV1 consumer from SLM-265 evidence before --policy targeted can be implemented."
+  ],
+  "no_claims": [
+    "No 10k, 100k, or 1M corpus was published.",
+    "No coverage-targeted policy was run.",
+    "No training experiment (3-seed 10k comparison) was run.",
+    "No checkpoint, promotion, or ship claim is made."
+  ],
+  "version_stamp": {
+    "stamp_schema": "version_stamp/v1",
+    "code_commit": "d3d0a4f836c88aeb28e4911d27a0db386a680e6b",
+    "code_dirty": true,
+    "components": {
+      "data.progspec.corpus_stream": "v1"
+    },
+    "stamped_at": "2026-07-25T07:54:51.479309+00:00"
+  }
+}

docs/design/iter-slm267-uniform-saturation-20260725.md

@@ -0,0 +1,31 @@
+# SLM-267 (VSD2-02) uniform-policy first reading — inconclusive
+
+Status: local smoke evidence; non-promotable; no checkpoint, training, or ship
+claim.
+
+Full narrative and follow-up scope:
+[`compiler-inverted-program-data.md`](compiler-inverted-program-data.md).
+Machine-readable evidence:
+[`iter-slm267-uniform-saturation-20260725.json`](iter-slm267-uniform-saturation-20260725.json).
+
+| Shard | Fresh start | Calls | New accepted | Cumulative accepted | Disposition |
+| --- | --- | ---: | ---: | ---: | --- |
+| 1 | yes | 334 | 334 | 334 | `wall_clock_stopped` |
+| 2 (resume) | no | 324 | 0 | 334 | `wall_clock_stopped` |
+
+- Config: all 54 pinned components, `max_depth=3`, `max_width=3`,
+  `seed=0`, `--max-wall-minutes 2.5` per shard (under the repo's
+  `MAX_RUN_MINUTES` cap).
+- Candidate grid size for this config: **1,781** — verified near-constant
+  across `max_depth`/`max_width` up to (10, 8) → 1,793, confirming the grid
+  is dominated by pairwise/prop-target coverage, not depth/width.
+- Zero verifier rejections (334/334 accepted as Silver).
+- **Resume-throughput blocker**: shard 2 spent its entire capped budget
+  replaying shard 1's calls and landed 0 net-new records — the current MVP
+  resume mechanism does not scale across many small shards at measured
+  verifier cost (~0.45–0.46s/call).
+
+Disposition: `inconclusive` for VSD-H7a. Neither 10k reached nor the current
+1,781-candidate grid exhausted. Two concrete blockers are filed for the next
+increment: a real candidate-cursor resume (not replay) and a wider
+`GeneratorConfig` candidate space.