Skip to content

DeltaWire preflight v6: preinstalled Gemini agent boundary - #115

Closed
bigboateng wants to merge 11 commits into
mainfrom
eval/deltawire-preflight-v6
Closed

DeltaWire preflight v6: preinstalled Gemini agent boundary#115
bigboateng wants to merge 11 commits into
mainfrom
eval/deltawire-preflight-v6

Conversation

@bigboateng

@bigboateng bigboateng commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

This experiment PR establishes the DeltaWire preflight v6 deterministic agent boundary and stops before any paid model call.

It preserves v1-v5 evidence, preinstalls and pins Gemini CLI in the task image, replaces Harbor's dynamic Gemini installation with a local verification-only custom agent, retains all three credential-free conformance trials, and freezes the resulting execution manifest.

This PR is intentionally an experiment/evidence branch. Raw jobs, logs, configs, locks, results, and receipts are retained here for review. A later clean integration PR should project only reusable implementation and compact evidence onto main.

ZCA slices

  1. Deterministic boundary: isolate Harbor/Gemini bootstrap from DeltaWire treatment behavior and prove the preinstalled runtime without a model or credential.
  2. Live proof: remains gated. The single paid D1 probe requires the exact later authorization phrase approve the paid v6 probe; the pair requires a second exact authorization phrase.

The immediate value is that a future paid failure can be attributed to treatment behavior rather than npm installation or agent bootstrap.

Why v6 exists

The v5 paid probe failed before agent execution because Harbor's official GeminiCli.install() dynamically ran apt/NVM/npm setup and timed out. V5 therefore established environment/artifact transport but did not test the model, DeltaWire lifecycle, or semantic output.

V6 keeps Harbor 0.20.0's official Gemini execution behavior while overriding only:

  • name() for an explicit custom-agent identity;
  • install() for local path, version, package, and image checks.

Official run, authentication, settings, skill, prompt, trajectory, ATIF, token, cost, and resume behavior remain inherited.

Changes

  • Added preinstalled-gemini-cli-v1 with structural tests proving only name and install are overridden.
  • Added a Linux/amd64 image pinned to the exact Node base manifest, Node 22.23.1, npm 10.9.8, Gemini CLI 0.51.0, Python, and one staged DeltaWire binary.
  • Added deterministic preinstalled-gemini-setup-receipt.v1 generation and validation.
  • Added independent Harbor process/trial normalization so process exit zero cannot hide a trial exception.
  • Added versioned range-large-v6, probe/pair tasks, contracts, schema, artifact gates, release gates, and a runner exposing only probe-v6 and pair.
  • Added 15 negative controls for missing/wrong executables, versions, package/image identity, malformed/misplaced receipts, dynamic installation, forbidden overrides, built-in agent use, credentials, models, and agent execution.
  • Added v6-only no-live-call GitHub Actions validation.
  • Retained all raw conformance evidence and froze 99 execution inputs in the v6 manifest.

No-model conformance results

Exactly three credential-free Harbor 0.20.0 install-only trials completed:

Trial Setup time Model Credential Agent execution Status
1 4.280944 s none absent none pass
2 4.319003 s none absent none pass
3 4.301793 s none absent none pass

All three produced identical canonical setup-receipt bytes:

a88d221e573265997797dcad31136aec0ed74a5bd37e50887595e5ea85cfa0d1

Harbor 0.20.0 disables verifier collect hooks under --install-only; it still automatically retained /logs/artifacts, including the setup receipt. The paid path separately requires the main collect hook, environment receipt, and successful artifact manifest.

Frozen identity

  • PR head: d889f7ebdb73044f0bed1abefbdc294cda203843
  • Manifest input commit: a4202620c388a3abce5e63d740986e366209b76e
  • Manifest SHA-256: ea5a4b4073604cd940f3f016cae390ae214b48f281fd30ed4581eedd4f99bc7a
  • Conformance report SHA-256: 6488387dbd1c4598be003aabc302ab9d37fe3d2c5eaab0d6a9263db6fad44f45
  • Image ID: sha256:589fce58d7ddf09910876afe0198bcee0086a840af6de7c08515bbce3e8367a7
  • DeltaWire binary SHA-256: e5198d15000e093a2a28e57ad4a093dde8f66bcab21462549ddf9114064c25f4
  • Harbor: 0.20.0
  • Model reserved for the future probe: google/gemini-3.1-pro-preview
  • Agent import: agents.preinstalled_gemini_cli:PreinstalledGeminiCli

The frozen VM dry-run verified every lock/hash and resolved exactly one D1 trial. Both live result roots were empty.

Validation

  • Full DeltaWire Go tests, race tests, 20x tests, vet, runtime-boundary and determinism checks passed.
  • Existing 12 canonical contracts and 96 mutation failures passed.
  • V1-v5 Git-object evidence locks passed.
  • V6 receipt, artifact, shell observation, fidelity, runner, binary parity, matrix, cleanliness, and no-live-CI checks passed.
  • Exact manifest-head workflow: https://github.com/operatorstack/intelligence-flow/actions/runs/29940053302
  • No Gemini credential or live provider call was used by conformance or CI.

Review focus

Please review whether:

  1. the custom agent truly preserves official Harbor run behavior;
  2. image and package identity are frozen strongly enough for the paid boundary;
  3. install-only artifact transport is classified correctly;
  4. the runner cannot expose historical probes, reruns, canaries, or full execution;
  5. the manifest and retained raw evidence are sufficient to authorize exactly one D1 probe;
  6. any additional deterministic test is required before the paid authorization phrase is given.

Authorization and readiness

No paid probe or pair has run. This PR does not authorize either.

{
  "READY_FOR_72": false,
  "probe_authorized": false,
  "probe_launched": false,
  "pair_launched": false,
  "six_run_canary_started": false,
  "full_run_started": false
}

A future passing D1 probe would prove only one end-to-end treatment path. It would not establish savings, cost effect, family-wide behavior, statistical significance, or readiness for 72 runs.

Current terminal experiment state (supersedes the earlier pre-live readiness text)

The paid v6 D1 probe passed at evidence commit 0dad5c01684b337f8a1bf42f17141f4789c95740 and produced content-addressed probe-release.v4 516be1dff108126fa15d6108b2720b80ed7fa39615a6698e52cd22270df6ac5e.

The separately authorized frozen range-large/r1 pair attempt is now terminal at evidence commit 273e977c. D1 completed with exact semantic output and passed setup, environment, artifact, activation, lifecycle, retained-artifact, and contract gates. It failed treatment fidelity only because the structured trajectory did not execute the mandatory pre-activation gemini skills list discovery call. The frozen runner therefore stopped before B0. No causal comparison or token-effect claim exists, and no rerun is permitted.

  • D1 trial: b3a68ff5-3215-4d29-a698-2b364ac261b9
  • Semantic result: exact ordered records 1–500
  • Treatment classification: attempted_failed
  • Failure class: missing_structured_pre_activation_discovery
  • B0: not launched
  • General DeltaWire validation: pass
  • V6-only validation: known unchanged frozen_negative_test_not_post_release_safe fixture failure
  • Credential and generic secret scan: zero matches
{
  "READY_FOR_72": false,
  "pair_attempt_consumed": true,
  "pair_complete": false,
  "pair_valid": false,
  "six_run_canary_started": false,
  "full_run_started": false
}

Superseded provenance

Reusable v6 runtime infrastructure and the Git-object-backed terminal evidence lock were cleanly integrated by PR #117, merge commit 82a7963bdd98ca41435e756fb4212b3b37b19420. This PR remains the immutable raw experiment provenance and is closed without merge or rebase.

@bigboateng

Copy link
Copy Markdown
Contributor Author

V6 paid probe evidence — PASS

The single authorized D1-only range-large-v6 probe ran once from frozen commit d889f7ebdb73044f0bed1abefbdc294cda203843. No pair, rerun, canary, full run, or merge was launched.

One-shot identity

  • Evidence commit: 0dad5c01
  • Trial ID: 338b89f9-ca55-45a7-a488-dbe64ba13e8f
  • Manifest SHA-256: ea5a4b4073604cd940f3f016cae390ae214b48f281fd30ed4581eedd4f99bc7a
  • Release ID: 516be1dff108126fa15d6108b2720b80ed7fa39615a6698e52cd22270df6ac5e
  • Release SHA-256: b4e7880d3024488516bfe559b09fc699204cef3b31c9e00f8c78583f21e6ccce
  • Output SHA-256: 9d205b57ab166d23c81927899f99b6ccece261c679c96979c58fbde975637776
  • Setup receipt SHA-256: a88d221e573265997797dcad31136aec0ed74a5bd37e50887595e5ea85cfa0d1

Gates

  • Harbor process exit: 0
  • Trial status: completed; exception: null
  • Agent/model identity: pass
  • Setup/environment/artifact manifest: pass
  • Structured skill discovery and activation: pass
  • Public contract access: pass
  • DeltaWire validate/render/check: pass
  • Retained config/schema/plan/state/output: pass
  • In-container and independent contract checks: pass
  • Semantic exact match: 1 for ordered records 1–500
  • End-to-end status: pass
  • Independent release recheck: pass
  • Exact credential and generic secret scans: pass
  • Pair root: empty

Usage diagnostics

  • Input tokens: 1,323,830
  • Output tokens: 20,011
  • Cache tokens: 1,086,095
  • Trial wall time: 376.437076 s
  • Agent setup: 4.38634 s

These are reported separately. No cost or savings claim is made.

Frozen validation limitation

The full DeltaWire validation passes. The frozen v6-only suite now reaches a pre-existing negative test that assumes probe-release.json is absent and requires the exact message probe-release.v4 missing. A real retained release invalidates that test premise. No frozen runner, test, workflow, manifest input, or runtime was modified to conceal this; it is recorded in evidence-validation.json and may make the evidence-head v6 check red.

{
  "READY_FOR_72": false,
  "pair_launched": false,
  "six_run_canary_started": false,
  "full_run_started": false,
  "pair_requires_explicit_review": true
}

The pair remains unauthorized unless the user later says exactly approve the frozen range-large/r1 pair.

@bigboateng

Copy link
Copy Markdown
Contributor Author

Evidence-head CI conclusions

Both v6 failures are identical: the frozen negative test invokes verify_release("missing") and expected the pre-release short-circuit probe-release.v4 missing. With the real release present, verification advances to reading the fabricated manifest path and raises FileNotFoundError: missing. All preceding v6 receipt, artifact, status, shell-observation, treatment-fidelity, inheritance, and negative-control tests pass.

No frozen input was modified to repair or hide this. The live release was independently reverified successfully and the limitation is retained in evidence-validation.json for the next review plan.

@bigboateng

Copy link
Copy Markdown
Contributor Author

Frozen pair attempt: stopped after D1 fidelity gate

Evidence commit: 273e977c

Exactly one live pair command was invoked. D1 completed with no Harbor exception, reward 1, and exact records 1–500. Environment, artifact transport, model/agent identity, skill activation, DeltaWire validate/render/check, retained artifacts, and both contract checks passed. The treatment receipt failed solely because structured pre-activation gemini skills list discovery was absent. The runner correctly withheld B0, so the attempt is consumed and no causal token comparison exists.

D1 usage: input 1,227,632; output 11,590; input+output 1,239,222; cache 979,337; Harbor-reported cost $0.8315374; wall time 245.613068 s. Cache is reported separately.

Full DeltaWire validation passed. The unchanged v6-only workflow is expected to retain the documented frozen_negative_test_not_post_release_safe failure. Secret scans found zero credential or generic-secret matches. READY_FOR_72=false; no rerun, B0, canary, full run, or merge occurred.

@bigboateng

Copy link
Copy Markdown
Contributor Author

Evidence-head CI

Evidence head: 273e977c2f61e3b3f31dd675ee3512ef9182a695

Both red runs pass evidence locks, environment, artifact, setup, trial normalization, shell-observation, and treatment-use tests before failing at the unchanged test_runner.py fabricated missing manifest path. This is the previously recorded frozen_negative_test_not_post_release_safe limitation; no frozen test or runner was changed.

@bigboateng

Copy link
Copy Markdown
Contributor Author

Reusable v6 infrastructure and the terminal evidence lock merged cleanly in PR #117 at 82a7963bdd98ca41435e756fb4212b3b37b19420. Closing this draft without merge; it remains immutable raw experiment provenance at evidence commit 273e977c2f61e3b3f31dd675ee3512ef9182a695.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant