feat(deltawire): representation optimizer (Stage 2) + pre-materialization safety gate

[bigboateng]

Summary

Adds DeltaWire's missing Stage 2 layer (per eval/docs/11-direct-optimal-spec-research-plan.md, GAPS.md #4 and #10): a deterministic representation-selection system. Given a direct spec it computes a feature vector, enumerates candidate semantic representations × transports, computes a Pareto frontier, and emits a deterministic recommendation with reason codes — then hands a safe spec to the unchanged supervisor.

DeltaWire stops being merely a format: it can now say "this is a high-fit finite relationship, use direct YAML" — or "this is trivial / irregular / structurally unsafe, do not use DeltaWire."

Architecture

direct spec (YAML | canonical JSON | compact JSON | typed tool args)
      ↓  transport normalization → one canonical spec SHA
representation profiler → feature vector (+ pre-materialization cardinality)
      ↓
candidate enumeration → {semantic form} × {transport}
      ↓
Pareto frontier → deterministic selector
      ↓  recommend / optional / not_recommended / abstain (+ reason codes)
[enforced safety gate: cardinality / output / plan / overflow — before expansion]
      ↓
existing supervisor (strategy-neutral, unchanged) → verified publication

Three concerns are kept strictly separated:

• Advisory/policy (internal/optimizer): recommended/optional/not_recommended are non-blocking; abstain is blocking (selection_blocking: true).
• Safety invariant (directspec.Preflight, enforced in the compile path): pre-materialization cardinality/output/plan/overflow bounds; hard failure → abstain + failure_class: resource_bound_exceeded.
• Supervision (internal/supervisor, untouched): only ever receives an already-selected admissible candidate. Recommendation is never encoded as FSM events.

What changed

Area	Change
internal/optimizer/ (new)	features, candidates, pareto, select, transport — feature vector, candidate metrics, non-dominated frontier (no weighted score), metric-driven selector
internal/directspec/guard.go (new)	Closed-form Cardinality, single-record EstimateOutputBytes, and the enforced Preflight safety gate
internal/directspec/parser.go	LoadJSON + LoadTyped share one finalize (validate → canonicalize → SHA) tail so all transports normalize identically
internal/cli/supervise.go	Preflight wired in before Compile — rejects oversized/unbounded specs before any row expansion (closes GAPS #4)
internal/cli/optimize.go (new)	deltawire optimize <spec> [--format markdown]

Design decisions (confirmed)

• Python = policy-excluded comparison class. Modeled as unsafe_program with implemented:false, policy_admissible:false, arbitrary_code:true, reason codes not_implemented/unsafe_program_disallowed/outside_generation_boundary. Never generated, executed, or selected. Byte metrics are null, never fabricated. No-executable-code invariant intact.
• Advisory + real safety gate. The optimizer advises; the compile-path gate enforces. Supervisor boundary unchanged.

Applicability — outcomes emerge from metrics (case names never hard-coded)

Case	Decision
range-piecewise, matrix-conditional, matrix/range pure	recommended
variants	optional
explicit rows	optional or not_recommended (by amplification)
trivial (card ≤ 1 or low amplification)	not_recommended
irregular (author-declared)	abstain
unsafe large product / overflow / output-blowup	abstain + resource_bound_exceeded

Verification — all gates pass

• go test ./..., -race, -count=20, go vet — pass
• bash scripts/validate.sh → EXIT 0 (clean state), git diff --check clean
• 14 canonical cases (every outcome), 105 deterministic mutation failures, transport equivalence (identical SHA ×4 transports), semantic equivalence (identical output bytes across representations + under transport), repeated-byte determinism, and a supervise-level gate test
• Zero model/provider calls — pure Go, deterministic

Example (deltawire optimize, range-piecewise, 300 records):

{"decision":"recommended","selection_blocking":false,
 "selected_form":"direct_relation","selected_transport":"compact_json",
 "reason_codes":["regular_finite_relation","high_amplification","piecewise_conditional"]}

Note (pre-existing, not from this PR)

A second consecutive scripts/validate.sh run trips eval/scripts/v8/validate.py: build_and_manifest.sh stages a freshly-built (gitignored) linux binary, but the script checks it against a pin (e5198d15…) frozen to older code — which even a clean main build (3a4f2f17…) no longer matches. A clean single run (as CI does) passes. The frozen evidence pin was intentionally left untouched.