Skip to content

eval(deltawire): prepare 72-run preflight and semantic oracles - #99

Closed
bigboateng wants to merge 30 commits into
mainfrom
eval/deltawire-72-preflight
Closed

eval(deltawire): prepare 72-run preflight and semantic oracles#99
bigboateng wants to merge 30 commits into
mainfrom
eval/deltawire-72-preflight

Conversation

@bigboateng

@bigboateng bigboateng commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Superseded experiment provenance. This conflicted PR is retained only for historical and raw evidence. The clean integration is PR #106. The final v4 evidence commit is c200dc0e.

Purpose

This lineage established fail-closed DeltaWire treatment delivery and semantic verification before any larger evaluation. It never authorized or launched the 72-run experiment.

Evidence summary

Overhead control: matrix-medium/r1

One real adjacent B0/D1 pair completed and both arms passed. D1 had the skill configured but did not activate or invoke DeltaWire. D1 reported 39,027 input-plus-output tokens versus 24,397 for B0, approximately 60% more. This pair is excluded from confirmatory estimates.

Historical probe v1

Delivery passed, but semantic verification failed: 999 {id,value} records were produced instead of 500 ordered {index} records. Derived status: delivery pass, semantic fail, end-to-end fail.

External-contract probe v2

Exactly one D1-only probe ran. Semantic verification passed, but end-to-end delivery failed because the structured transcript lacked gemini skills list evidence and the frozen artifact list did not retain .deltawire/config.json. No pair was launched and v2 was not rerun.

Recovery probe v3

Exactly one D1-only probe ran. Semantic exact match, discovery, activation, lifecycle, hashes, retention, and independent contract verification passed. The frozen parser rejected which deltawire as binary-path proof, so delivery and end-to-end status failed. No release or pair was launched.

Measurement-recovery probe v4

Exactly one D1-only range-large-v4 probe ran on tbench-c4d using the frozen v4 manifest and binary.

  • Harbor 0.20.0 completed without exception with reward 1.000 in 261.440381 seconds.
  • Semantic exact match passed for all 500 canonical {index} records.
  • Structured discovery and activation, public-contract access, DeltaWire lifecycle invocation, config/schema/plan/state/output retention, schema integrity, and both contract checks passed.
  • The deterministic verifier-phase environment generator produced a passing receipt: expected path/realpath, executable regular non-symlink binary, exact version and zero exit, binary SHA, and config/schema/public-contract hashes all passed.
  • Harbor could not retrieve .deltawire/environment-receipt.json from the main task container after the verifier wrote it in the verifier container. The independently retained environment receipt was therefore missing.
  • Fail-closed status: environment fail, treatment-use pass, semantic pass, end-to-end fail.
  • Diagnostics: 934,798 input tokens, 10,344 output tokens, 750,963 cache tokens, and 261.440381 seconds wall time.

No probe-release.v2 was created. The authorized range-large/r1 pair was not launched. Under the one-shot stop condition, v4 was not repaired or rerun and no v5 was created.

Validation and publication

Readiness

READY_FOR_72=false. Remaining blockers include the incomplete six-run canary, no final confirmatory manifest, and no explicit 72-run approval. The v4 pair and 72-run experiment were not launched. No savings, cost, statistical-significance, or family-wide claim is made.

bigboateng and others added 30 commits July 21, 2026 18:43
- Register `gemini` adapter parsing the synchronous `BeforeTool` hook

- Update Pitot CLI `hook` and `doctor` commands to recognize `gemini`

- Add `e2e_gemini_cli_test.sh` to satisfy the supervisory control build gate

- Update E2E and windtunnel testing coverage to ensure parity with Cursor/Claude/Codex boundaries

- Append mandatory release note and sync public distribution JSON
Implements the standalone DeltaWire v1 Go CLI.
- Strict JSON parsing and Draft 2020-12 schema validation
- Matrix, rows, and variant generation
- Managed outputs and strict path bounds
- Full CI/validation scripts and tests
…ipeline

Replace brittle mock-oriented canary plumbing with runner-owned Harbor
artifact contracts, grounded extraction/oracle checks, and readiness that
proves one real pair while keeping READY_FOR_72 false.

Co-authored-by: Cursor <cursoragent@cursor.com>
@bigboateng

Copy link
Copy Markdown
Contributor Author

Superseded by the clean integration PR #106, now merged as e3e085d. This PR remains immutable experiment provenance. Final v4 evidence: c200dc0. Passing v4 validation: https://github.com/operatorstack/intelligence-flow/actions/runs/29924758152. No post-hoc v4 release was created; READY_FOR_72 remains false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant