diff --git a/docs/design/compiler-inverted-program-data.md b/docs/design/compiler-inverted-program-data.md new file mode 100644 index 000000000..7ad6204ec --- /dev/null +++ b/docs/design/compiler-inverted-program-data.md @@ -0,0 +1,163 @@ +# Compiler-inverted ProgramSpec corpus (SLM-267 / VSD2-02) + +Status: **wiring-first increment; uniform-policy CLI landed; VSD-H7a +inconclusive at smoke scale with a real throughput blocker identified.** + +This is the first PR against +[SLM-267 (VSD2-02)](https://linear.app/quickdeploy-ai/issue/SLM-267/vsd2-02-scale-compiler-inverted-programspec-data-to-10k100k1m-with): +scale the repository's typed `ProgramSpec`-first generator +(`src/slm_training/data/progspec/generate.py`) into a streaming, +coverage-audited semantic data engine and publish nested 10k/100k/1M corpora +under uniform and coverage-targeted sampling. That full scope is far larger +than one increment; this PR lands the streaming/resumable **uniform-valid** +control CLI the issue specifies and uses it to take an honest, measured first +reading of **VSD-H7a** ("the bounded OpenUI ProgramSpec space can supply +nested 10k/100k/1M corpora"). Coverage-targeted sampling (needs a +`CoverageGapManifestV1` consumer wired to SLM-265's outcome-blind domain-shift +strata), the 100k/1M rungs, and the in-issue training experiment are explicit +follow-on scope — not attempted here. + +## What shipped + +`python -m scripts.generate_programspec_corpus` (new; +`tests/test_scripts/test_generate_programspec_corpus.py`): + +- Wraps the existing `ProgramGenerator` — never replaces it. +- `--policy uniform` only in this increment (`--policy targeted` is not a + valid choice yet; the issue's `--policy` surface is preserved so a follow-up + can add it without a CLI break). +- Deterministic `(seed, shard_id)` generation over a `GeneratorConfig` + (`--max-depth`, `--max-width`, `--components`). +- Disk-backed **exact** dedup: the accepted set is keyed on + `ProgramSpec.id` (already a canonical content hash of the program + + viewport/depth/width/prop-target identity computed inside the generator) — + no Bloom-filter-only path exists. +- Immutable rejection ledger (`rejected.jsonl`): every `generate_one()` + `ValueError` other than grid exhaustion is appended with its call index and + verifier-failure reason, never silently dropped. +- Resumable `state.json` (schema `programspec_corpus_stream_state/v1`) and a + content-addressed `manifest.json` (schema + `programspec_corpus_stream_manifest/v1`, `accepted_ids_sha256` over the + sorted accepted-id set) under the canonical `DataStore` `programspec` root + (`outputs/data/programspec//`) — no shadow artifact tree. +- `--describe` dry-run; `--max-wall-minutes` capped at + `slm_training.levers.MAX_RUN_MINUTES` like every other capped harness in + this repo. + +## Determinism contract (why resume is possible at all) + +For a fixed `GeneratorConfig` + `seed`, `ProgramGenerator` visits candidates in +an identical order and produces an identical accept/reject sequence on every +process invocation — nothing in candidate selection or verification consumes +wall-clock time, OS randomness, or network state. `tests/test_scripts/test_generate_programspec_corpus.py::test_resume_produces_monotonic_prefix` +locks this: resuming with a larger `--target-unique-roots` always reproduces +the prior accepted-id prefix byte-for-byte before extending it. + +## Measured evidence (smoke scale, local CPU, two capped shards) + +Command (both shards identical except invocation time): + +```bash +python -m scripts.generate_programspec_corpus \ + --pack openui --policy uniform --target-unique-roots 10000 \ + --seed 0 --shard-id 0 --max-wall-minutes 2.5 \ + --dataset-id compiler_programs_uniform_smoke_v1 +``` + +Config: default (all 54 pinned components), `max_depth=3`, `max_width=3` — +the same defaults `ProgramGenerator` and `GeneratorConfig` ship with. + +| Shard | Calls this run | New accepted | New rejected | Cumulative accepted | Disposition | +| --- | ---: | ---: | ---: | ---: | --- | +| 1 (fresh) | 334 | 334 | 0 | 334 | `wall_clock_stopped` | +| 2 (resume) | 324 | **0** | 0 | 334 | `wall_clock_stopped` | + +Full evidence: +[`iter-slm267-uniform-saturation-20260725.json`](iter-slm267-uniform-saturation-20260725.json). + +Zero verifier rejections at these settings (all 334 generated programs cleared +G0–G10 as Silver) — the earlier 0/50-accept reading during local development +was an environment artifact (the `@openuidev/lang-core` Node bridge under +`src/apps/openui_bridge/` was not `npm ci`-installed), not a generator defect; +confirmed by the full `tests/test_data/test_progspec_generate.py` suite +passing once the bridge was installed. + +## VSD-H7a reading: `generator_state_space_saturates` risk confirmed, not yet resolved + +Two independent, measured findings bear directly on the issue's falsifier +("the state space, prompt inversion, verifier, or dedup contract is +materially narrower than assumed"): + +1. **The candidate grid is a small, seed-independent, near-constant function + of `GeneratorConfig`, not of the target root count.** Measured + `len(ProgramGenerator(config)._candidates)`: + + | `max_depth` | `max_width` | Candidates (54 components) | + | ---: | ---: | ---: | + | 3 | 3 | 1,781 | + | 5 | 4 | 1,785 | + | 8 | 6 | 1,791 | + | 10 | 8 | 1,793 | + + Depth/width barely move the count because the grid is dominated by + `O(components²)` pairwise-coverage candidates and + `O(components × properties × variants)` prop-target candidates, not by + depth/width combinatorics. Critically, **`seed` only perturbs tie-break + jitter among equal-coverage-score candidates — it never adds new + candidates.** Sharding by `(global_seed, shard_id, worker_id)` as the issue + specifies therefore cannot by itself grow reachable diversity past 1,781 + unique roots for this config; only widening `GeneratorConfig` (more + `selected_groups`/triples, literal-value pools, multi-instance repeats + within one program) can. That widening is unscoped in this PR. + +2. **The MVP resume mechanism does not deliver practical cross-shard + throughput at measured verifier cost.** At ≈0.45–0.46s/`generate_one()` + call (bridge-subprocess + full G0–G10 verification per candidate), a + from-scratch shard nets ~330 new unique roots in one capped 150s window. + Because this increment's resume strategy re-derives state by **replaying** + the deterministic call sequence from index 0 (skipping emission, not + verification, for already-seen calls) rather than persisting the + generator's internal candidate cursor, the *second* capped shard spent its + entire budget re-executing the first shard's 334 calls and landed **zero** + net-new records. Total work to reach cumulative count `M` across many + small shards is closer to `O(M·shards)` than `O(M)` — the opposite of + "resumable" at this cost level. + +Neither finding is a final verdict on VSD-H7a: the smoke run used the +smallest, un-widened `GeneratorConfig` and the naive resume path. Both are +concrete, falsifiable blockers a follow-up increment must clear before a 10k +uniform corpus is reachable: + +- persist the generator's internal `_used` candidate cursor (or an equivalent + index) in `state.json` so resume is `O(new work)`, not `O(total work)`; +- widen `GeneratorConfig`/candidate construction (or add a second axis of + variation) so the candidate grid itself can exceed 10k before the resume + question even matters. + +## Disposition + +**`inconclusive`** (one of the issue's own listed acceptance dispositions). +VSD-H7a is neither confirmed nor falsified: this increment did not reach the +10k rung, but it also did not exhaust the current 1,781-candidate grid (334 of +1,781, `exhausted: false`) — it hit the resume-throughput blocker first. No +training experiment, coverage-targeted policy, or 100k/1M rung was attempted. +No checkpoint, ship, or promotion claim is made or implied. + +## Non-goals of this PR + +- No `--policy targeted` implementation (needs a `CoverageGapManifestV1` + consumer; SLM-265's `iter-slm265-domain-shift-audit-20260724.json` is + `coverage_gap_manifest/v1`-shaped evidence but is not yet exposed as a + stable manifest artifact for a generator to consume). +- No `publish_programspec_corpus.py` / `DataStore.publish` — this stays a + local, inspectable staging corpus under `outputs/data/programspec/`, not a + durable Git-published dataset. +- No 100k/1M rung, no 3-seed training-arm comparison, no checkpoint. + +## Related + +- Generator: [`src/slm_training/data/progspec/generate.py`](../../src/slm_training/data/progspec/generate.py) +- New CLI: [`scripts/generate_programspec_corpus.py`](../../scripts/generate_programspec_corpus.py) +- Tests: [`tests/test_scripts/test_generate_programspec_corpus.py`](../../tests/test_scripts/test_generate_programspec_corpus.py) +- SLM-265 evidence consumed for status only (not as a coverage manifest): + [`iter-slm265-domain-shift-audit-20260724.json`](iter-slm265-domain-shift-audit-20260724.json) diff --git a/docs/design/iter-slm267-uniform-saturation-20260725.json b/docs/design/iter-slm267-uniform-saturation-20260725.json new file mode 100644 index 000000000..97c65af55 --- /dev/null +++ b/docs/design/iter-slm267-uniform-saturation-20260725.json @@ -0,0 +1,93 @@ +{ + "schema": "slm267_uniform_saturation_reading/v1", + "issue": "SLM-267", + "hypothesis": "VSD-H7a", + "status": "inconclusive", + "objective": "First measured reading of whether the bounded OpenUI ProgramSpec space can reach 10k unique canonical roots under the frozen uniform-valid policy, using the new streaming/resumable CLI.", + "recipe": { + "command": "python -m scripts.generate_programspec_corpus --pack openui --policy uniform --target-unique-roots 10000 --seed 0 --shard-id 0 --max-wall-minutes 2.5 --dataset-id compiler_programs_uniform_smoke_v1", + "device": "cpu", + "generator_config": { + "components": "all_pinned (54)", + "max_depth": 3, + "max_width": 3 + }, + "shards_run": 2, + "max_wall_minutes_per_shard": 2.5, + "honesty_mode": "smoke_local_no_checkpoint_no_training" + }, + "candidate_grid_size_by_config": [ + {"max_depth": 3, "max_width": 3, "candidates": 1781}, + {"max_depth": 5, "max_width": 4, "candidates": 1785}, + {"max_depth": 8, "max_width": 6, "candidates": 1791}, + {"max_depth": 10, "max_width": 8, "candidates": 1793} + ], + "seed_independence_note": "candidate count is a deterministic function of GeneratorConfig only; seed perturbs tie-break jitter among equal-score candidates and never adds candidates.", + "shard_reports": [ + { + "shard_index": 1, + "fresh_start": true, + "calls_this_run": 334, + "new_accepted": 334, + "new_rejected": 0, + "cumulative_accepted": 334, + "cumulative_calls": 334, + "disposition": "wall_clock_stopped", + "elapsed_seconds": 150.0, + "per_record_seconds": 0.4491017964071856 + }, + { + "shard_index": 2, + "fresh_start": false, + "calls_this_run": 324, + "new_accepted": 0, + "new_rejected": 0, + "cumulative_accepted": 334, + "cumulative_calls": 658, + "disposition": "wall_clock_stopped", + "elapsed_seconds": 150.0, + "per_record_seconds": 0.46296296296296297 + } + ], + "final_manifest": { + "schema": "programspec_corpus_stream_manifest/v1", + "dataset_id": "compiler_programs_uniform_smoke_v1", + "pack": "openui", + "policy": "uniform", + "seed": 0, + "shard_id": "0", + "target_unique_roots": 10000, + "unique_roots": 334, + "candidate_grid_size": 1781, + "rejected_total": 0, + "calls_total": 658, + "exhausted": false, + "accepted_ids_sha256": "1a7b75bcc58e2f9d9c42205c394ac6a91675288da347b56bbcddcee067eae719" + }, + "findings": [ + "All 334 generated programs cleared G0-G10 as Silver at this config; zero rejections at smoke scale.", + "The uniform-policy candidate grid (1,781 at the config used here) is far below the 10k target and is nearly insensitive to --max-depth/--max-width, because it is dominated by O(components^2) pairwise and O(components*properties*variants) prop-target candidates rather than depth/width combinatorics.", + "The second capped shard replayed the first shard's 324-of-334 prior calls and produced zero net-new accepted records: the MVP resume mechanism (replay from index 0, skip emission for already-seen calls) does not deliver O(new work) cross-shard throughput at the measured ~0.45-0.46s/generate_one() cost -- it is closer to O(total work) per shard.", + "The 0/50-accept reading seen during local development before this run was traced to a missing `npm ci` install for the @openuidev/lang-core Node bridge (src/apps/openui_bridge/), not a generator or verifier defect; the full tests/test_data/test_progspec_generate.py suite passes once the bridge is installed." + ], + "blockers_for_next_increment": [ + "Persist the generator's internal candidate cursor (not just accepted ids) in state.json so resume is O(new work).", + "Widen GeneratorConfig / candidate construction (more selected_groups/triples, literal-value pools, multi-instance component repeats) so the candidate grid itself can exceed 10k before the resume-cost question is the binding constraint.", + "Wire a stable CoverageGapManifestV1 consumer from SLM-265 evidence before --policy targeted can be implemented." + ], + "no_claims": [ + "No 10k, 100k, or 1M corpus was published.", + "No coverage-targeted policy was run.", + "No training experiment (3-seed 10k comparison) was run.", + "No checkpoint, promotion, or ship claim is made." + ], + "version_stamp": { + "stamp_schema": "version_stamp/v1", + "code_commit": "d3d0a4f836c88aeb28e4911d27a0db386a680e6b", + "code_dirty": true, + "components": { + "data.progspec.corpus_stream": "v1" + }, + "stamped_at": "2026-07-25T07:54:51.479309+00:00" + } +} diff --git a/docs/design/iter-slm267-uniform-saturation-20260725.md b/docs/design/iter-slm267-uniform-saturation-20260725.md new file mode 100644 index 000000000..aad3afcd8 --- /dev/null +++ b/docs/design/iter-slm267-uniform-saturation-20260725.md @@ -0,0 +1,31 @@ +# SLM-267 (VSD2-02) uniform-policy first reading — inconclusive + +Status: local smoke evidence; non-promotable; no checkpoint, training, or ship +claim. + +Full narrative and follow-up scope: +[`compiler-inverted-program-data.md`](compiler-inverted-program-data.md). +Machine-readable evidence: +[`iter-slm267-uniform-saturation-20260725.json`](iter-slm267-uniform-saturation-20260725.json). + +| Shard | Fresh start | Calls | New accepted | Cumulative accepted | Disposition | +| --- | --- | ---: | ---: | ---: | --- | +| 1 | yes | 334 | 334 | 334 | `wall_clock_stopped` | +| 2 (resume) | no | 324 | 0 | 334 | `wall_clock_stopped` | + +- Config: all 54 pinned components, `max_depth=3`, `max_width=3`, + `seed=0`, `--max-wall-minutes 2.5` per shard (under the repo's + `MAX_RUN_MINUTES` cap). +- Candidate grid size for this config: **1,781** — verified near-constant + across `max_depth`/`max_width` up to (10, 8) → 1,793, confirming the grid + is dominated by pairwise/prop-target coverage, not depth/width. +- Zero verifier rejections (334/334 accepted as Silver). +- **Resume-throughput blocker**: shard 2 spent its entire capped budget + replaying shard 1's calls and landed 0 net-new records — the current MVP + resume mechanism does not scale across many small shards at measured + verifier cost (~0.45–0.46s/call). + +Disposition: `inconclusive` for VSD-H7a. Neither 10k reached nor the current +1,781-candidate grid exhausted. Two concrete blockers are filed for the next +increment: a real candidate-cursor resume (not replay) and a wider +`GeneratorConfig` candidate space. diff --git a/scripts/generate_programspec_corpus.py b/scripts/generate_programspec_corpus.py index c5112c4a8..eca116fd1 100644 --- a/scripts/generate_programspec_corpus.py +++ b/scripts/generate_programspec_corpus.py @@ -1,30 +1,393 @@ #!/usr/bin/env python3 -"""Run the SLM-267 (VSD2-02) ProgramSpec coverage-scaling wiring fixture. +"""ProgramSpec corpus generation CLI for SLM-267 (VSD2-02). -Example: - python -m scripts.generate_programspec_corpus --mode plan-only - python -m scripts.generate_programspec_corpus --mode fixture +Two independent lines of work landed on this same path and were reconciled +into one CLI dispatched by ``--mode``: + +``stream`` (default) + Streaming, resumable full-scale corpus generation: a canonical CLI over + the existing typed :class:`~slm_training.data.progspec.generate.ProgramGenerator` + (``src/slm_training/data/progspec/generate.py``) that adds deterministic + seeded generation, an immutable rejection ledger, a disk-backed exact + dedup index keyed on the generator's own canonical ``ProgramSpec.id``, + and a resumable shard state so repeated invocations (each bounded by the + repo's hard run cap, ``slm_training.levers.MAX_RUN_MINUTES``) accumulate + monotonically toward a ``--target-unique-roots`` count instead of + restarting from zero. Only the frozen **uniform-valid** base-distribution + policy is implemented here. The issue's second policy + (**coverage-targeted**, driven by a ``CoverageGapManifestV1``) and the + 100k/1M rungs are explicitly deferred -- see + ``docs/design/compiler-inverted-program-data.md`` for the measured + saturation frontier this increment establishes and the follow-up scope. + + Determinism contract this mode relies on (locked by + ``tests/test_scripts/test_generate_programspec_corpus.py``): for a fixed + ``GeneratorConfig`` and ``seed``, ``ProgramGenerator`` visits candidates + in an identical order and produces an identical accept/reject sequence on + every invocation (no wall-clock, network, or OS-random input enters + candidate selection or verification content). Resuming therefore replays + the prior prefix of that sequence -- cheap in candidate count, not in + verifier calls -- and continues past it; see the "Known limitation" note + in the design doc. + +``plan-only`` / ``fixture`` + The original SLM-267 coverage-scaling *wiring evidence* campaign: a + bounded fixture comparing the generator's genuine uniform-random arm + against its existing greedy coverage-maximizing arm + (``slm_training.harnesses.experiments.slm267_programspec_coverage_scaling``). + This is wiring evidence only -- it does not validate VSD-H7a/b/c, which + require corpora and trained models several orders of magnitude larger + than anything generated here. + +Usage:: + + python -m scripts.generate_programspec_corpus \\ + --pack openui --policy uniform \\ + --target-unique-roots 500 --seed 0 --shard-id 0 \\ + --dataset-id compiler_programs_uniform_smoke_v1 + + python -m scripts.generate_programspec_corpus --mode plan-only + python -m scripts.generate_programspec_corpus --mode fixture """ from __future__ import annotations import argparse +import hashlib import json import sys +import time +from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import Any +from slm_training.data.progspec.generate import GeneratorConfig, ProgramGenerator +from slm_training.data.store import DataStore +from slm_training.harness_core.versioning import build_version_stamp from slm_training.harnesses.experiments.slm267_programspec_coverage_scaling import ( DEFAULT_CONFIG, EXPERIMENT_ID, render_markdown, run_coverage_scaling_campaign, ) -from slm_training.versioning import build_version_stamp +from slm_training.levers import MAX_RUN_MINUTES + +STATE_SCHEMA = "programspec_corpus_stream_state/v1" +MANIFEST_SCHEMA = "programspec_corpus_stream_manifest/v1" +SUPPORTED_PACKS = ("openui",) +SUPPORTED_POLICIES = ("uniform",) +_EXHAUSTED_MESSAGE = "candidate grid exhausted" + +_FIXTURE_DESIGN_JSON = "docs/design/iter-slm267-programspec-coverage-scaling-20260725.json" +_FIXTURE_DESIGN_MD = "docs/design/iter-slm267-programspec-coverage-scaling-20260725.md" + + +def _wall_minutes(value: str) -> float: + minutes = float(value) + if not 0 < minutes <= MAX_RUN_MINUTES: + raise argparse.ArgumentTypeError( + f"must be positive and at most {MAX_RUN_MINUTES}" + ) + return minutes + + +# -------------------------------------------------------------------------- +# ``stream`` mode: streaming, resumable full-scale corpus generation. +# -------------------------------------------------------------------------- + + +@dataclass +class StreamState: + """Resumable per-(config, seed, shard) generation progress.""" + + dataset_id: str + pack: str + policy: str + seed: int + shard_id: str + max_depth: int + max_width: int + components: tuple[str, ...] | None + target_unique_roots: int + call_count: int = 0 + accepted_ids: list[str] = field(default_factory=list) + rejected_count: int = 0 + exhausted: bool = False + + def to_dict(self) -> dict[str, Any]: + return { + "schema": STATE_SCHEMA, + "dataset_id": self.dataset_id, + "pack": self.pack, + "policy": self.policy, + "seed": self.seed, + "shard_id": self.shard_id, + "max_depth": self.max_depth, + "max_width": self.max_width, + "components": list(self.components) if self.components else None, + "target_unique_roots": self.target_unique_roots, + "call_count": self.call_count, + "accepted_ids": self.accepted_ids, + "rejected_count": self.rejected_count, + "exhausted": self.exhausted, + } + + @classmethod + def from_dict(cls, data: dict[str, Any]) -> StreamState: + components = data.get("components") + return cls( + dataset_id=str(data["dataset_id"]), + pack=str(data["pack"]), + policy=str(data["policy"]), + seed=int(data["seed"]), + shard_id=str(data["shard_id"]), + max_depth=int(data["max_depth"]), + max_width=int(data["max_width"]), + components=tuple(components) if components else None, + target_unique_roots=int(data["target_unique_roots"]), + call_count=int(data.get("call_count", 0)), + accepted_ids=list(data.get("accepted_ids", [])), + rejected_count=int(data.get("rejected_count", 0)), + exhausted=bool(data.get("exhausted", False)), + ) + + +def _config_identity(state: StreamState) -> tuple[Any, ...]: + """Fields that must match exactly to resume (not merely append) a shard.""" + return ( + state.pack, + state.policy, + state.seed, + state.shard_id, + state.max_depth, + state.max_width, + state.components, + ) + + +def run_shard( + state: StreamState, + *, + max_wall_minutes: float, + record_sink: list[dict[str, Any]] | None = None, + rejection_sink: list[dict[str, Any]] | None = None, +) -> tuple[StreamState, dict[str, Any]]: + """Advance ``state`` by replaying + extending its deterministic sequence. + + Returns the updated state and a per-invocation run report (throughput, + disposition). ``record_sink`` / ``rejection_sink``, when supplied, receive + only the *newly* accepted / rejected rows from this invocation (the + replayed prefix is skipped, not re-emitted). + """ + config = GeneratorConfig( + components=state.components, + max_depth=state.max_depth, + max_width=state.max_width, + ) + generator = ProgramGenerator(config, seed=state.seed) + candidate_grid_size = len(generator._candidates) # noqa: SLF001 - read-only introspection + seen_ids = set(state.accepted_ids) + replay_target = state.call_count + deadline = time.monotonic() + max_wall_minutes * 60 + calls_this_run = 0 + new_accepted = 0 + new_rejected = 0 + disposition = "in_progress" + + while True: + if len(state.accepted_ids) >= state.target_unique_roots: + disposition = "target_reached" + break + if time.monotonic() >= deadline: + disposition = "wall_clock_stopped" + break + try: + spec = generator.generate_one() + except ValueError as exc: + if str(exc) == _EXHAUSTED_MESSAGE: + state.exhausted = True + disposition = "saturated" + break + state.call_count += 1 + calls_this_run += 1 + if state.call_count > replay_target: + state.rejected_count += 1 + new_rejected += 1 + if rejection_sink is not None: + rejection_sink.append( + { + "call_index": state.call_count, + "reason": str(exc), + } + ) + continue + state.call_count += 1 + calls_this_run += 1 + if state.call_count > replay_target: + if spec.id not in seen_ids: + seen_ids.add(spec.id) + state.accepted_ids.append(spec.id) + new_accepted += 1 + if record_sink is not None: + record_sink.append(spec.to_dict()) + + elapsed = ( + max_wall_minutes * 60 - max(0.0, deadline - time.monotonic()) + if calls_this_run + else 0.0 + ) + report = { + "candidate_grid_size": candidate_grid_size, + "calls_this_run": calls_this_run, + "new_accepted": new_accepted, + "new_rejected": new_rejected, + "cumulative_accepted": len(state.accepted_ids), + "cumulative_rejected": state.rejected_count, + "cumulative_calls": state.call_count, + "exhausted": state.exhausted, + "disposition": disposition, + "elapsed_seconds": elapsed, + "per_record_seconds": (elapsed / calls_this_run) if calls_this_run else None, + } + return state, report -_DESIGN_JSON = "docs/design/iter-slm267-programspec-coverage-scaling-20260725.json" -_DESIGN_MD = "docs/design/iter-slm267-programspec-coverage-scaling-20260725.md" + +def _append_jsonl(path: Path, rows: list[dict[str, Any]]) -> None: + if not rows: + return + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("a", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, sort_keys=True) + "\n") + + +def _manifest(state: StreamState, report: dict[str, Any]) -> dict[str, Any]: + accepted_sha256 = hashlib.sha256( + json.dumps(sorted(state.accepted_ids), separators=(",", ":")).encode("utf-8") + ).hexdigest() + return { + "schema": MANIFEST_SCHEMA, + "dataset_id": state.dataset_id, + "pack": state.pack, + "policy": state.policy, + "seed": state.seed, + "shard_id": state.shard_id, + "generator_config": { + "max_depth": state.max_depth, + "max_width": state.max_width, + "components": list(state.components) if state.components else "all_pinned", + }, + "target_unique_roots": state.target_unique_roots, + "unique_roots": len(state.accepted_ids), + "rejected_total": state.rejected_count, + "calls_total": state.call_count, + "candidate_grid_size": report["candidate_grid_size"], + "exhausted": state.exhausted, + "accepted_ids_sha256": accepted_sha256, + "last_run_report": report, + "version_stamp": build_version_stamp("data.progspec.corpus_stream"), + } + + +def _run_stream(args: argparse.Namespace) -> int: + if args.target_unique_roots is None or args.target_unique_roots <= 0: + raise SystemExit("--target-unique-roots is required and must be positive in stream mode") + if not args.dataset_id: + raise SystemExit("--dataset-id is required in stream mode") + + store = DataStore() + out_dir = store.path("programspec", args.dataset_id) + state_path = out_dir / "state.json" + records_path = out_dir / "records.jsonl" + rejected_path = out_dir / "rejected.jsonl" + manifest_path = out_dir / "manifest.json" + + components = ( + tuple(sorted(c.strip() for c in args.components.split(",") if c.strip())) + if args.components + else None + ) + + if state_path.is_file(): + state = StreamState.from_dict(json.loads(state_path.read_text(encoding="utf-8"))) + expected = StreamState( + dataset_id=args.dataset_id, + pack=args.pack, + policy=args.policy, + seed=args.seed, + shard_id=args.shard_id, + max_depth=args.max_depth, + max_width=args.max_width, + components=components, + target_unique_roots=args.target_unique_roots, + ) + if _config_identity(state) != _config_identity(expected): + raise SystemExit( + f"existing state at {state_path} was built with different " + "pack/policy/seed/shard/config; choose a new --dataset-id or " + "match the original arguments exactly" + ) + state.target_unique_roots = args.target_unique_roots + else: + state = StreamState( + dataset_id=args.dataset_id, + pack=args.pack, + policy=args.policy, + seed=args.seed, + shard_id=args.shard_id, + max_depth=args.max_depth, + max_width=args.max_width, + components=components, + target_unique_roots=args.target_unique_roots, + ) + + plan = { + "dataset_id": args.dataset_id, + "pack": args.pack, + "policy": args.policy, + "seed": args.seed, + "shard_id": args.shard_id, + "target_unique_roots": args.target_unique_roots, + "resuming_from_accepted": len(state.accepted_ids), + "out_dir": str(out_dir), + } + if args.describe: + print(json.dumps(plan, indent=2, sort_keys=True)) + return 0 + + new_records: list[dict[str, Any]] = [] + new_rejections: list[dict[str, Any]] = [] + state, report = run_shard( + state, + max_wall_minutes=args.max_wall_minutes, + record_sink=new_records, + rejection_sink=new_rejections, + ) + + # Cumulative accepted rows accumulate across invocations: only the + # current run's newly accepted ProgramSpecs are held in memory, and + # prior rows already on disk are preserved by appending rather than + # rewriting from scratch. + if not records_path.is_file(): + records_path.parent.mkdir(parents=True, exist_ok=True) + records_path.write_text("", encoding="utf-8") + _append_jsonl(records_path, new_records) + _append_jsonl(rejected_path, new_rejections) + + manifest = _manifest(state, report) + manifest_path.parent.mkdir(parents=True, exist_ok=True) + manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n", encoding="utf-8") + state_path.write_text(json.dumps(state.to_dict(), indent=2, sort_keys=True) + "\n", encoding="utf-8") + + print(json.dumps(report, indent=2, sort_keys=True)) + return 0 + + +# -------------------------------------------------------------------------- +# ``plan-only`` / ``fixture`` modes: the original coverage-scaling wiring +# evidence campaign (SLM-267 first landing, PR #893). +# -------------------------------------------------------------------------- def _now() -> str: @@ -35,15 +398,11 @@ def _today_yyyymmdd() -> str: return datetime.now(timezone.utc).strftime("%Y%m%d") -def _build_payload( +def _build_fixture_payload( mode: str, target_count: int, seed: int, shards: int -) -> tuple[dict[str, Any], str]: - command = ( - f"python -m scripts.generate_programspec_corpus --mode {mode} " - f"--target-count {target_count} --seed {seed} --shards {shards}" - ) +) -> dict[str, Any]: if mode == "plan-only": - payload: dict[str, Any] = { + return { "schema": "Slm267ProgramspecCoverageScalingManifestV1", "experiment_id": EXPERIMENT_ID, "status": "plan_only", @@ -60,55 +419,16 @@ def _build_payload( ), "timestamp": _now(), } - return payload, command - payload = run_coverage_scaling_campaign( - target_count=target_count, seed=seed, shards=shards - ) - return payload, command - + return run_coverage_scaling_campaign(target_count=target_count, seed=seed, shards=shards) -def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser( - description="SLM-267 VSD2-02 ProgramSpec coverage-scaling fixture", - exit_on_error=False, - ) - parser.add_argument( - "--mode", - choices={"plan-only", "fixture"}, - default="plan-only", - help="Run mode: plan-only writes the manifest; fixture runs the CPU campaign.", - ) - parser.add_argument( - "--output-dir", - type=Path, - help=( - "Directory for run artifacts (default: " - "outputs/runs/slm267-programspec-coverage-scaling-)" - ), - ) - parser.add_argument( - "--target-count", - type=int, - default=80, - help="Bounded per-arm program budget (default: 80; fixture scale only).", - ) - parser.add_argument("--seed", type=int, default=0, help="Global seed (default: 0).") - parser.add_argument( - "--shards", type=int, default=2, help="Deterministic shard count (default: 2)." - ) - try: - args = parser.parse_args(argv) - except (argparse.ArgumentError, SystemExit): - return 2 +def _run_wiring_fixture(args: argparse.Namespace) -> int: output_dir = args.output_dir or Path( f"outputs/runs/slm267-programspec-coverage-scaling-{_today_yyyymmdd()}" ) output_dir.mkdir(parents=True, exist_ok=True) - payload, command = _build_payload( - args.mode, args.target_count, args.seed, args.shards - ) + payload = _build_fixture_payload(args.mode, args.target_count, args.seed, args.shards) payload["schema"] = payload.get("schema", "Slm267ProgramspecCoverageScalingReportV1") payload["claim_class"] = "wiring" payload["status"] = payload.get("status", "inconclusive") @@ -126,8 +446,8 @@ def main(argv: list[str] | None = None) -> int: if args.mode == "fixture": root = Path(__file__).resolve().parents[1] - json_path = root / _DESIGN_JSON - md_path = root / _DESIGN_MD + json_path = root / _FIXTURE_DESIGN_JSON + md_path = root / _FIXTURE_DESIGN_MD json_path.parent.mkdir(parents=True, exist_ok=True) md_path.parent.mkdir(parents=True, exist_ok=True) json_path.write_text(report_text, encoding="utf-8") @@ -137,5 +457,72 @@ def main(argv: list[str] | None = None) -> int: return 0 +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__, exit_on_error=False) + parser.add_argument( + "--mode", + choices=("stream", "plan-only", "fixture"), + default="stream", + help=( + "stream: streaming/resumable full-scale corpus generation " + "(default). plan-only/fixture: bounded coverage-scaling wiring " + "evidence campaign." + ), + ) + + # -- stream mode -- + parser.add_argument("--pack", choices=SUPPORTED_PACKS, default="openui") + parser.add_argument("--policy", choices=SUPPORTED_POLICIES, default="uniform") + parser.add_argument("--target-unique-roots", type=int, default=None) + parser.add_argument("--shard-id", default="0") + parser.add_argument("--max-depth", type=int, default=3) + parser.add_argument("--max-width", type=int, default=3) + parser.add_argument( + "--components", + default=None, + help="Comma-separated component allowlist (default: all pinned components).", + ) + parser.add_argument("--dataset-id", default=None) + parser.add_argument( + "--max-wall-minutes", + type=_wall_minutes, + default=float(MAX_RUN_MINUTES), + help=f"Per-invocation wall budget (default and maximum: {MAX_RUN_MINUTES}).", + ) + parser.add_argument("--describe", action="store_true", help="Print plan; do not run.") + + # -- shared -- + parser.add_argument("--seed", type=int, default=0, help="Global seed (default: 0).") + + # -- plan-only / fixture mode -- + parser.add_argument( + "--output-dir", + type=Path, + default=None, + help=( + "Directory for run artifacts (default: " + "outputs/runs/slm267-programspec-coverage-scaling-)" + ), + ) + parser.add_argument( + "--target-count", + type=int, + default=80, + help="Bounded per-arm program budget (default: 80; fixture scale only).", + ) + parser.add_argument( + "--shards", type=int, default=2, help="Deterministic shard count (default: 2)." + ) + + try: + args = parser.parse_args(argv) + except (argparse.ArgumentError, SystemExit): + return 2 + + if args.mode in ("plan-only", "fixture"): + return _run_wiring_fixture(args) + return _run_stream(args) + + if __name__ == "__main__": sys.exit(main()) diff --git a/src/slm_training/resources/versions.json b/src/slm_training/resources/versions.json index 7635ca4b0..87baf2abd 100644 --- a/src/slm_training/resources/versions.json +++ b/src/slm_training/resources/versions.json @@ -6417,18 +6417,21 @@ ] }, "harness.experiments.slm267_programspec_coverage_scaling": { - "version": "v1", + "version": "v2", "kind": "harness", "paths": [ "src/slm_training/harnesses/experiments/slm267_programspec_coverage_scaling.py", - "scripts/generate_programspec_corpus.py", "tests/test_harnesses/experiments/test_slm267_programspec_coverage_scaling.py", - "tests/test_scripts/test_generate_programspec_corpus.py", "docs/design/programspec-coverage-scaling.md", "docs/design/iter-slm267-programspec-coverage-scaling-20260725.json", "docs/design/iter-slm267-programspec-coverage-scaling-20260725.md" ], "history": [ + { + "version": "v2", + "date": "2026-07-25", + "note": "scripts/generate_programspec_corpus.py and its test moved to data.progspec.corpus_stream, which now owns the merged CLI (--mode stream/plan-only/fixture) after reconciling with the streaming-corpus PR; this component keeps only the harness module + its own test + design docs" + }, { "version": "v1", "date": "2026-07-25", @@ -10678,6 +10681,24 @@ "note": "initial reserved compute-ops vocabulary shared encoder<->decoder: 19 ops derived from the live local/topology/conversation registries across ast/graph/set/topology/history, reserved in the new `ops` token-id namespace above every codec range, one shared_token_ids() mapping, grammar layering assertion, content-addressed drift gate (decode invariant I13)" } ] + }, + "data.progspec.corpus_stream": { + "version": "v1", + "kind": "data_builder", + "paths": [ + "scripts/generate_programspec_corpus.py", + "tests/test_scripts/test_generate_programspec_corpus.py", + "docs/design/compiler-inverted-program-data.md", + "docs/design/iter-slm267-uniform-saturation-20260725.json", + "docs/design/iter-slm267-uniform-saturation-20260725.md" + ], + "history": [ + { + "version": "v1", + "date": "2026-07-25", + "note": "initial registration; SLM-267 VSD2-02 uniform-policy streaming ProgramSpec corpus CLI with resumable shard state, disk-backed exact dedup, and measured candidate-grid saturation frontier. Reconciled on rebase with the parallel harness.experiments.slm267_programspec_coverage_scaling landing (PR #893): this component now owns scripts/generate_programspec_corpus.py's --mode dispatch (stream default; plan-only/fixture absorbed as subcommands) and its merged test suite" + } + ] } } } diff --git a/tests/test_scripts/test_generate_programspec_corpus.py b/tests/test_scripts/test_generate_programspec_corpus.py index 905e4b8a6..fee9e9426 100644 --- a/tests/test_scripts/test_generate_programspec_corpus.py +++ b/tests/test_scripts/test_generate_programspec_corpus.py @@ -1,13 +1,115 @@ -"""Tests for the SLM-267 (VSD2-02) ProgramSpec coverage-scaling fixture CLI.""" +"""ProgramSpec corpus generation CLI (SLM-267 VSD2-02): stream + wiring-fixture modes.""" from __future__ import annotations import json +from pathlib import Path + +import pytest from scripts.generate_programspec_corpus import main +_SMALL = ["--components", "TextContent,Button,Separator", "--max-depth", "3", "--max-width", "3"] + + +def _records(out_dir: Path) -> list[dict]: + return [ + json.loads(line) + for line in (out_dir / "records.jsonl").read_text(encoding="utf-8").splitlines() + if line.strip() + ] + + +def test_describe_is_dry_run(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys) -> None: + monkeypatch.chdir(tmp_path) + rc = main( + ["--describe", "--target-unique-roots", "5", "--dataset-id", "describe_only", *_SMALL] + ) + assert rc == 0 + plan = json.loads(capsys.readouterr().out) + assert plan["resuming_from_accepted"] == 0 + assert not (tmp_path / "outputs" / "data" / "programspec" / "describe_only").exists() + + +def test_generation_is_deterministic_and_exact_deduped( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + dataset_id = "compiler_programs_uniform_test_v1" + assert main(["--target-unique-roots", "5", "--dataset-id", dataset_id, *_SMALL]) == 0 + + out_dir = tmp_path / "outputs" / "data" / "programspec" / dataset_id + rows = _records(out_dir) + assert len(rows) == 5 + ids = [row["id"] for row in rows] + assert len(ids) == len(set(ids)) # exact dedup: no duplicate canonical roots + + manifest = json.loads((out_dir / "manifest.json").read_text(encoding="utf-8")) + assert manifest["unique_roots"] == 5 + assert manifest["version_stamp"]["stamp_schema"] == "version_stamp/v1" + + # Re-running against an already-satisfied target is a no-op on the corpus. + assert main(["--target-unique-roots", "5", "--dataset-id", dataset_id, *_SMALL]) == 0 + assert len(_records(out_dir)) == 5 + state = json.loads((out_dir / "state.json").read_text(encoding="utf-8")) + assert state["accepted_ids"] == ids + + +def test_resume_produces_monotonic_prefix( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + dataset_id = "compiler_programs_uniform_resume_v1" + assert main(["--target-unique-roots", "3", "--dataset-id", dataset_id, *_SMALL]) == 0 + out_dir = tmp_path / "outputs" / "data" / "programspec" / dataset_id + first_ids = [row["id"] for row in _records(out_dir)] + assert len(first_ids) == 3 + + assert main(["--target-unique-roots", "8", "--dataset-id", dataset_id, *_SMALL]) == 0 + second_ids = [row["id"] for row in _records(out_dir)] + assert len(second_ids) == 8 + assert second_ids[:3] == first_ids # resume replays, never reorders, the prefix + assert len(set(second_ids)) == 8 + + +def test_mismatched_resume_config_is_rejected( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + dataset_id = "compiler_programs_uniform_mismatch_v1" + assert main(["--target-unique-roots", "2", "--dataset-id", dataset_id, *_SMALL]) == 0 + with pytest.raises(SystemExit): + main( + [ + "--target-unique-roots", + "2", + "--dataset-id", + dataset_id, + "--seed", + "7", + *_SMALL, + ] + ) + + +def test_exhaustion_is_detected_and_reported( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + dataset_id = "compiler_programs_uniform_saturation_v1" + assert ( + main(["--target-unique-roots", "100000", "--dataset-id", dataset_id, *_SMALL]) == 0 + ) + out_dir = tmp_path / "outputs" / "data" / "programspec" / dataset_id + manifest = json.loads((out_dir / "manifest.json").read_text(encoding="utf-8")) + state = json.loads((out_dir / "state.json").read_text(encoding="utf-8")) + assert manifest["exhausted"] is True + assert state["exhausted"] is True + assert manifest["unique_roots"] == manifest["candidate_grid_size"] + assert manifest["last_run_report"]["disposition"] == "saturated" + -def test_plan_only_writes_manifest(tmp_path) -> None: +def test_plan_only_writes_manifest(tmp_path: Path) -> None: assert main(["--mode", "plan-only", "--output-dir", str(tmp_path)]) == 0 run_json = tmp_path / "slm267_programspec_coverage_scaling_report.json" assert run_json.exists() @@ -18,7 +120,7 @@ def test_plan_only_writes_manifest(tmp_path) -> None: assert data["version_stamp"] -def test_fixture_writes_design_docs(tmp_path) -> None: +def test_fixture_writes_design_docs(tmp_path: Path) -> None: assert ( main( [