Skip to content

DRB-12: Classify missing stage-1 and S1-S11 baseline/scorecard artifacts #198

Description

@metaphorics

Problem

Several expected performance baselines and scorecards are referenced by live policy and manifest but do not exist in the working tree. They must be classified as required, intentionally fail-closed, or abandoned, with an explicit owner and lifecycle for each.

Exact missing artifacts

  • bench/baselines/stage1.json: expected by bench/compiler-rules.toml:139 and the default in bench/stage1_regression_guard.rs:24.
  • bench/scorecards/stage1.json: expected by the default in bench/stage1_regression_guard.rs:28.
  • perf/baselines/s1.json through perf/baselines/s11.json: expected by perf/benchmarks.toml:26,36,46,56,66,76,86,96,106,116,127 for slices s1..s11. Only perf/baselines/s0.json is present; s11 is selected = true and is exercised by crates/bamts-verification/src/perf.rs:2620 perf_budget_rejects_slice_without_baseline.

Scope of change

  • For each missing path, classify as: (a) required and must be materialized/captured from an approved measurement, (b) intentionally fail-closed placeholder, or (c) abandoned and references removed.
  • Update the manifest, gate ledger, and consumers (bench/stage1_regression_guard.rs, perf_budget, bamts-verification perf_guard/tests) to match the classification.
  • No fabricated baseline: any newly created baseline must come from an approved measurement run with host conditions and evidence recorded.

Source-of-truth predecessors

  • Performance architecture decision: DRB-8 (which decides whether bench/ and perf/ are coexisting or unified into one surface).
  • Baseline policy and host conditions: the surviving bench/compiler-rules.toml or perf/hosts/bh1.toml (after DRB-8).
  • Scorecard generation: bamts-verification perf_guard::evaluate_stage1_guard_from_paths and perf_budget compare.
  • Manifest/benchmark inventory: perf/benchmarks.toml and verification/catalog-inputs.json (benchmarks catalog).

Acceptance

  • Every path in bench/compiler-rules.toml, bench/stage1_regression_guard.rs, perf/benchmarks.toml, and perf_guard.rs that names a baseline/scorecard has a documented owner and lifecycle (required/fail-closed/removed).
  • No expected = "perf/baselines/s*.json" or baseline = "bench/baselines/*.json" points at a hand-fabricated or unmeasured file.
  • perf_budget compare and bamts-verification completion verify --leaf E5.4 produce a deterministic failure mode (missing-baseline error or pass) rather than an unresolved file-not-found panic.
  • The manifest or gate ledger explicitly records which S1-S11 slices are selected and which are intentionally unmeasured.

Blockers

  • DRB-8 (dual performance authority) must decide whether bench/ and perf/ are the active surface(s).
  • DRB-6 (generated cache/workspace residue) may purge ignored boundary fixtures but does not own baseline lifecycle.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wayfinder:taskDecision prerequisite ticket

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions