eval(deltawire): prepare 72-run preflight and semantic oracles - #99
Closed
bigboateng wants to merge 30 commits into
Closed
eval(deltawire): prepare 72-run preflight and semantic oracles#99bigboateng wants to merge 30 commits into
bigboateng wants to merge 30 commits into
Conversation
- Register `gemini` adapter parsing the synchronous `BeforeTool` hook - Update Pitot CLI `hook` and `doctor` commands to recognize `gemini` - Add `e2e_gemini_cli_test.sh` to satisfy the supervisory control build gate - Update E2E and windtunnel testing coverage to ensure parity with Cursor/Claude/Codex boundaries - Append mandatory release note and sync public distribution JSON
Implements the standalone DeltaWire v1 Go CLI. - Strict JSON parsing and Draft 2020-12 schema validation - Matrix, rows, and variant generation - Managed outputs and strict path bounds - Full CI/validation scripts and tests
… and actual validations
…g checks, and commit fresh canary jobs
…task instructions
…ipeline Replace brittle mock-oriented canary plumbing with runner-owned Harbor artifact contracts, grounded extraction/oracle checks, and readiness that proves one real pair while keeping READY_FOR_72 false. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Superseded by the clean integration PR #106, now merged as e3e085d. This PR remains immutable experiment provenance. Final v4 evidence: c200dc0. Passing v4 validation: https://github.com/operatorstack/intelligence-flow/actions/runs/29924758152. No post-hoc v4 release was created; READY_FOR_72 remains false. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This lineage established fail-closed DeltaWire treatment delivery and semantic verification before any larger evaluation. It never authorized or launched the 72-run experiment.
Evidence summary
Overhead control:
matrix-medium/r1One real adjacent B0/D1 pair completed and both arms passed. D1 had the skill configured but did not activate or invoke DeltaWire. D1 reported 39,027 input-plus-output tokens versus 24,397 for B0, approximately 60% more. This pair is excluded from confirmatory estimates.
Historical probe v1
Delivery passed, but semantic verification failed: 999
{id,value}records were produced instead of 500 ordered{index}records. Derived status: delivery pass, semantic fail, end-to-end fail.External-contract probe v2
Exactly one D1-only probe ran. Semantic verification passed, but end-to-end delivery failed because the structured transcript lacked
gemini skills listevidence and the frozen artifact list did not retain.deltawire/config.json. No pair was launched and v2 was not rerun.Recovery probe v3
Exactly one D1-only probe ran. Semantic exact match, discovery, activation, lifecycle, hashes, retention, and independent contract verification passed. The frozen parser rejected
which deltawireas binary-path proof, so delivery and end-to-end status failed. No release or pair was launched.Measurement-recovery probe v4
Exactly one D1-only
range-large-v4probe ran ontbench-c4dusing the frozen v4 manifest and binary.1.000in 261.440381 seconds.{index}records..deltawire/environment-receipt.jsonfrom the main task container after the verifier wrote it in the verifier container. The independently retained environment receipt was therefore missing.No
probe-release.v2was created. The authorizedrange-large/r1pair was not launched. Under the one-shot stop condition, v4 was not repaired or rerun and no v5 was created.Validation and publication
5a8ee779: run 29923012991.55b8c499: run 29923193058.c200dc0e: run 29924758152.Readiness
READY_FOR_72=false. Remaining blockers include the incomplete six-run canary, no final confirmatory manifest, and no explicit 72-run approval. The v4 pair and 72-run experiment were not launched. No savings, cost, statistical-significance, or family-wide claim is made.