Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions PROJECT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,10 +142,13 @@ validation.
- Linux and macOS lack live platform evidence.
- The accepted economic results are calibrations, not a powered corpus or
general savings claim. The public v4 benchmark manifest is answer-free and
synthetic; the offline protocol and synthetic evaluator fixture are complete,
but real private material, a non-synthetic validated PowerPlan, and a
provider run are absent. App provider execution remains fail-closed until an
isolated executor/broker consumes only ArmLaunch. This does not prove
synthetic. Deterministic power planning, exact schedule/observation binding,
fail-closed paired analysis, and the two-sided 95% BCa report path are
implemented and offline validated. The checked-in plan is reproducible from
explicitly synthetic calibration observations and remains permanently
claim-ineligible. Real private material, a non-synthetic validated PowerPlan,
and a provider run are absent. App provider execution remains fail-closed
until an isolated executor/broker consumes only ArmLaunch. This does not prove
filesystem ACLs or process isolation; callers must keep private bundles
unmounted and inaccessible to the runner identity.
- No public beta, support channel, or compatibility window exists.
Expand Down
10 changes: 7 additions & 3 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ maintainers; no external bundle path is present in public manifests or arm
launch projections. A real sealed bundle and validated production PowerPlan
are absent from this checkout, so offline preflight remains fail-closed. The
synthetic bundle is complete for evaluator tests but cannot enable a provider
run. A future caller must keep private evaluator material unmounted and
inaccessible to the runner identity; this protocol does not claim ACL or
process-isolation proof.
run. `router-cache/synthetic-calibration-observations.jsonl` deterministically
reproduces the checked-in synthetic PowerPlan, while
`router-cache/synthetic-final-observations.jsonl` exercises the exact paired
holdout contract and its fail-closed synthetic boundary. None of these
artifacts is economic evidence. A future caller must keep private evaluator
material unmounted and inaccessible to the runner identity; this protocol does
not claim ACL or process-isolation proof.
4 changes: 3 additions & 1 deletion benchmarks/corpus/router-cache/campaign.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"schema": "needle.multi-task-campaign/2",
"schedule_digest": "b3:c853b4ed84d8e9119013d788a7c769c63fd77f63a947706fbe4c128b65ed51d5",
"schedule_digest": "b3:3ae9b089ba4c7bee7b58ad562a5797872377ef2d986fba30d75651cc567210dd",
"task_ids": [],
"paid_arms": [
"frontier_direct",
"needle_miss"
Expand All @@ -15,6 +16,7 @@
"native_subagent",
"escalation"
],
"repetitions_per_task": 0,
"automatic_retries": false,
"statistical_claim": false,
"bootstrap_resamples": 10000,
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/corpus/router-cache/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@
"next_pilot_path": "minimal-live-pilot.json",
"next_pilot_digest": "b3:18a434227eb9abf2381eb0a65348f0dba66301ccba7dbdeb57c7933b078180cb",
"campaign_path": "campaign.json",
"campaign_digest": "b3:082239af1ebe785555801af74c817c5e49c143759cd3e62ef6607f54dadf5e1d",
"campaign_digest": "b3:a434abd47f654b222576e60c29f1640fd311d7fa81fa66419018fb55b008329b",
"schedule_path": "schedule.json",
"schedule_digest": "b3:c853b4ed84d8e9119013d788a7c769c63fd77f63a947706fbe4c128b65ed51d5",
"schedule_digest": "b3:3ae9b089ba4c7bee7b58ad562a5797872377ef2d986fba30d75651cc567210dd",
"power_plan_path": "power-plan.json",
"power_plan_digest": "b3:8605f040792744907aa94d3ce017a211b774c16521dbe744acb7a1b587647630",
"power_plan_digest": "b3:0cfe45b5cca814731619776de15572959720cd576db2c0a8d8cf0bf8d0b561e6",
"sealed_bundle_schema": "needle.sealed-oracle-index/1",
"sealed_bundle_digest": "b3:3750e933fa1226fed1892dc4ba3ddad4f84fbf6bd58135d8f26355103dc681c7",
"tasks": [
Expand Down
29 changes: 15 additions & 14 deletions benchmarks/corpus/router-cache/power-plan.json
Original file line number Diff line number Diff line change
@@ -1,32 +1,33 @@
{
"schema": "needle.power-plan/1",
"plan_id": "synthetic-router-cache-v1",
"schema": "needle.power-plan/2",
"plan_id": "power-plan-1d6bfc9443debec0",
"manifest_digest": "b3:8be00a443b7e1a57813d8dbe25e6ea22a988f7536bc0f159fee8a58f0fa6cd7a",
"campaign_digest": "b3:647300dac4ba3c304b4052a114f1e80b0eb214e278fd4415bbd576c369652864",
"calibration_input_digest": "b3:4d27a8427bd3655d59e88c37efadcccfca4a7d82cd4a4324708180aea31289c0",
"estimator_revision": "issue-7-structural-v1",
"campaign_commitment": "b3:647300dac4ba3c304b4052a114f1e80b0eb214e278fd4415bbd576c369652864",
"calibration_input_digest": "b3:1d6bfc9443debec09dab40fdd8b72bd78e6e5e5bf01075def337b31f92c4a104",
"estimator_revision": "needle.paired-log-ratio-power/1",
"alpha_basis_points": 500,
"target_power_basis_points": 9000,
"routes": [
{
"route": "locate_implementation",
"baseline_arm": "frontier_direct",
"treatment_arm": "needle_miss",
"pair_key": "task_id:repetition",
"observed_log_ratio_mean": -0.4,
"observed_log_ratio_stddev": 0.2,
"required_pairs": 1
"pair_key": "corpus_digest:campaign_commitment:task_id:route:split:repetition:pair_seed",
"observed_log_ratio_mean": -0.4337502838523616,
"observed_log_ratio_stddev": 0.10900099103037066,
"required_pairs": 3
},
{
"route": "trace_state_flow",
"baseline_arm": "frontier_direct",
"treatment_arm": "needle_miss",
"pair_key": "task_id:repetition",
"observed_log_ratio_mean": -0.4,
"observed_log_ratio_stddev": 0.2,
"required_pairs": 1
"pair_key": "corpus_digest:campaign_commitment:task_id:route:split:repetition:pair_seed",
"observed_log_ratio_mean": -0.4337502838523616,
"observed_log_ratio_stddev": 0.10900099103037066,
"required_pairs": 3
}
],
"validated": true,
"synthetic": true
"synthetic": true,
"artifact_digest": "b3:88cadb0c43e0ac1a39566f5c76fa1a64949b2ab88bdb592aaaafc76b897b22fa"
}
Loading