docs: plan direct and optimal DeltaWire specification research

Author: bigboateng

labs/20-deltawire/eval/docs/11-direct-optimal-spec-research-plan.md

@@ -0,0 +1,130 @@
+# DeltaWire research plan: direct and optimal specification
+
+## Purpose
+
+Determine how an agent can describe a dataset relationship without authoring executable generation code, while preserving exactness and using fewer model tokens than both the current DeltaWire JSON plan and an unrestricted coding-agent baseline.
+
+ZCA separates two research slices:
+
+1. **Direct specification:** the agent states the required dataset relationship; a system-owned compiler produces and verifies the artifact.
+2. **Optimal specification:** the system selects the least expensive admissible representation or backend for that relationship.
+
+This is a product-interface investigation, not a larger benchmark or research-infrastructure program.
+
+## Evidence baseline
+
+The six-trial isolated `range-large` comparison established:
+
+- All six final artifacts were exact: 500 ordered records with only integer `index` values 1–500.
+- Both trials that genuinely completed the DeltaWire lifecycle were exact: 2/2.
+- DeltaWire treatment fidelity was 2/3; one D1 agent ignored the assigned plan and generated the artifact directly.
+- The valid JSON plan was 673 bytes and deterministically rendered 6,892 bytes, a 10.24x byte expansion.
+- The two valid D1/B0 comparisons used 1.543x and 1.073x the baseline input-plus-output tokens. Their median ratio was 1.308x, or 30.8% overhead.
+- Valid D1 required 10–12 tool calls; isolated B0 required 3–5.
+- B0 authored executable generation logic: two Bash loops and one Node.js loop. It did not use Python or DeltaWire.
+
+Accuracy is therefore proven only for this simple range: 6/6 final outputs and 2/2 genuine DeltaWire outputs. Earlier evidence showed that a structurally valid `matrix-large` plan could still be semantically wrong, so accuracy is not yet established across families.
+
+The current bottleneck is not deterministic expansion. It is the agent-facing acquisition path: skill activation, reference discovery, repeated reads, ownership rules, and editing an implementation-oriented JSON plan.
+
+## Stage 1: direct specification
+
+### Research question
+
+Can the agent express the desired record relationship directly, without loops or backend-specific generator instructions, in fewer interactions than both current JSON-plan authoring and arbitrary code generation?
+
+### Proposed boundary
+
+The agent supplies a declarative dataset contract containing only:
+
+- output format and path;
+- record schema and key;
+- finite domains or explicit examples;
+- relations such as ranges, products, variants, and exceptions;
+- derived fields and piecewise rules;
+- count, uniqueness, coverage, and ordering requirements.
+
+The agent must not supply Bash, Python, JavaScript, Go, SQL, or another executable program. A system-owned translator normalizes the contract, proves computable cardinality where possible, produces the existing DeltaWire plan, and invokes the existing validator, renderer, checker, and semantic oracle.
+
+Start with four falsifiable examples:
+
+1. A range with piecewise derived values.
+2. A Cartesian authorization matrix with conditional expected outcomes.
+3. A base record with missing, null, empty, malformed, and boundary variants.
+4. An irregular dataset that must be rejected or explicitly abstained from.
+
+Do not add multiple backends in this stage. Compile only to the current DeltaWire plan so the experiment isolates the agent-facing specification.
+
+### Experiment
+
+For each regular example, compare three arms under the same model and semantic oracle:
+
+- unrestricted agent-authored code;
+- current DeltaWire JSON plan;
+- direct declarative specification.
+
+Measure semantic exactness, treatment completion, input/output/cache tokens, tool calls, repair turns, wall time, specification bytes, generated-plan bytes, and artifact bytes. Keep null metrics null and report every failed outcome.
+
+### Stage-1 acceptance
+
+- Every accepted direct specification has statically computable cardinality and deterministic ordering.
+- Canonical and mutation fixtures prove exact semantics, not only schema validity.
+- No agent-authored executable code reaches the generation boundary.
+- Treatment completion is 3/3 on each high-fit family before cross-family claims.
+- Direct specification uses fewer tool calls and fewer median input-plus-output tokens than current JSON-plan authoring.
+- Token savings versus unrestricted code are reported only where both arms are exact.
+
+## Stage 2: optimal specification
+
+### Research question
+
+Given a validated dataset relationship, which admissible representation minimizes total generation cost without weakening determinism, safety, or verification?
+
+Define optimization subject to correctness:
+
+```text
+minimize agent tokens + repair turns + execution cost
+subject to exact semantic verification,
+           deterministic output,
+           no agent-authored executable code,
+           bounded expansion,
+           retained provenance
+```
+
+The system may evaluate equivalent internal realizations such as a native DeltaWire plan, a system-generated streaming program, or literal materialization. These are compiler outputs, not agent inputs. If Python is added later, it is generated and owned by the compiler; current DeltaWire does not accept Python scripts.
+
+### Selection model
+
+For each candidate representation, compute or estimate:
+
+- exact cardinality;
+- specification and generated-artifact bytes;
+- projected memory and streaming behavior;
+- backend capability coverage;
+- verification completeness;
+- expected agent interaction cost;
+- deterministic reproducibility.
+
+Reject ambiguous or unbounded models. Prefer the lowest-cost candidate that preserves the complete verification contract. Abstain when literal data or a repository-native fixture is simpler than a generated model.
+
+### Experiment
+
+Use scale points of 500, 10,000, and 100,000 records plus increasing relational complexity. Compare direct specification, the selected optimal representation, current JSON plans, and unrestricted code. Separate simple ranges from matrices, derived piecewise data, and variants so output size is not confused with reasoning complexity.
+
+### Stage-2 acceptance
+
+- Backend selection is deterministic for identical normalized input.
+- Every selected backend passes the same independent semantic oracle.
+- Selection never chooses executable generation when a safer representation has equal or lower measured cost.
+- High-fit workloads demonstrate lower median model tokens than current JSON plans and unrestricted code.
+- Low-fit workloads produce an explicit, correct abstention instead of forced DeltaWire use.
+
+## Deliverables and decision gates
+
+1. Freeze the six-trial JSON-plan evidence as the baseline.
+2. Publish a compact direct-specification grammar with the four examples and rejected ambiguity cases.
+3. Build one translator to the existing DeltaWire plan and validate it without changing rendering semantics.
+4. Run the three-arm direct-specification experiment and decide whether token overhead is removed.
+5. Only after Stage 1 succeeds, design and test optimal backend selection.
+
+The next implementation phase is Stage 1 only. It must not introduce arbitrary script input, multi-backend selection, a broad benchmark campaign, or claims beyond observed accuracy and token measurements.