Feature/0.0.0#3
Merged
Merged
Conversation
Record agent responses once against the real provider, then replay them so CI re-runs the gate without spending tokens. Because replay re-executes the AGENT against the recorded responses (not just the trace), a cost regression from an agent code change still surfaces — for free. - cassette/: a record/replay store keyed by (scenario, run, seq), JSONL on disk so it is diffable and committable. Create truncates (a recording is a fresh capture); Load reads for replay; Lookup misses surface agent divergence. - proxy/: Mode (live/record/replay) with Record/Replay setters. Record forwards and saves each response; replay serves from the cassette and never contacts the provider, regenerating an identical trace (latency/tokens preserved). A replay miss is a loud 502. seq is now assigned up front so record and replay key the cassette identically; usage extraction unified (extractUsage) so a buffered/replayed SSE body is costed exactly like the live stream. - augur run: -record / -replay flags (mutually exclusive). Session defaults to empty in record/replay so run ids are stable and the cassette keys line up; live runs keep the timestamp default. Tests: cassette round-trip / truncate / miss / malformed; proxy record- then-replay with the provider shut down (identical trace, zero provider calls via a failing transport), streaming record/replay, and replay miss → 502. Verified end-to-end: recorded 6 calls, killed the provider, replayed all 6, traces match.
Add multiplier knobs that re-cost a recorded trace under hypothetical
agentic cost drivers — no agent re-run, no tokens spent. This is the
systems/control-flow angle: the projection moves because a named driver
moved, not a flat fudge factor.
- cost/: factor out Breakdown (input/cached/output split) so the prompt
side can be scaled independently of the completion. Cost now sums a
Breakdown — behavior unchanged.
- aggregate/: Knobs{RetryRate, FanoutFactor, ContextGrowth} and
AggregateWithKnobs. Retries and fan-out scale the whole call
((1+retry)*fanout); context growth scales only the prompt
(input+cached). Per call: cost' = callMult*(growth*prompt + output).
The zero Knobs is the identity, so Aggregate delegates to it.
- project/ + gate/: a WhatIf description line surfaces the applied knobs
in the table and the Markdown report.
- augur project / augur gate: shared --retry-rate / --fanout /
--context-growth flags. Gate against a pessimistic scenario, not just
today's happy path.
Tests reconcile the knob math by hand (callMult and prompt-only context
growth), confirm context growth leaves a pure-output call unchanged,
cover Knobs identity, and verify the Breakdown components and that
Total equals Cost. Verified end-to-end: baseline vs what-if projection.
Self-hosted models have no per-token API price — you pay for an instance by the hour. Derive the effective token price with the standard benchmark-then-size calculation: $/Mtok = instance_$/hr x 1e6 / (tokens_per_sec x 3600 x utilization) Utilization is the honest knob: you pay for the box 24/7 but it's rarely saturated, and a half-idle instance doubles the effective price. - tco/: parse + validate tco.yaml (deployments with instance cost, throughput, optional utilization defaulting to 1.0) and derive a cost.Pricing where input/output/cached all carry the effective rate (self-hosted compute is token-type-agnostic). Plus a table renderer. - TCO is a pricing SOURCE, so the pipeline is untouched: a shared resolvePricing wires a -tco flag into aggregate/project/gate as an alternative to -pricing. New `augur tco` command shows the derived effective rates. - examples/tco.yaml documents the format. Tests reconcile the effective-rate math by hand ($36/hr, 2500 tok/s, 100% -> $4/Mtok; 50% util -> $8), cover validation (zero throughput, bad utilization, empty config) and the derived Pricing. Verified end-to-end: tco table + a trace re-costed against self-hosted pricing.
Ship Augur as a composite action that runs the gate on a pull request, posts the report as a PR comment, and fails the check when the projection is over budget. Combined with a committed cassette, the PR check spends no tokens. - action.yml: composite action — setup-go + build, optional cassette replay to regenerate the trace, `augur gate` (exit code captured so the comment posts before the job fails), upsert the report comment via actions/github-script (deduped by a hidden marker), then enforce the budget. Supports --pricing or --tco and pass-through extra-args (what-if knobs). Distinguishes an over-budget verdict (report written) from an internal error (no report) so CI failures are legible. - .github/workflows/ci.yml: Augur's own build/vet/test on push + PR. - examples/github-workflow.yml: a copy-paste consumer workflow. - .gitignore: ignore the built binary and generated trace/report artifacts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.