Feature/0.0.1#4
Merged
Merged
Conversation
The one piece the SPEC defers to a second language: learn completion
length from a recorded trace, then estimate cost for inputs you never
ran. Completion length is the quantity you cannot guess for an agent and
where the cost surprise lives, so it is the honest thing to model.
sidecar/ (Python, numpy):
- fit: per-model OLS output ~ input_tokens + run-template capture;
intercept-only fallback (method=mean) when the signal is
weak, reported honestly alongside R^2 and residual spread.
- report: fit quality, flagging weak/absent input->output signal.
- predict: output length (+ optional completion-cost) for one input.
- emit-trace: synthesise a predicted trace (rescaled inputs, sampled at
the residual spread so the p95 stays honest) that the Go
pipeline prices unchanged.
Coupling to the Go core is the JSONL trace file alone -- no RPC, no
shared library, no import either direction. Unlike --context-growth, the
sidecar feeds the larger prompt back through the output model, so a
bigger ask predicts a longer answer, not just a costlier prompt.
Verified end-to-end: recorded trace passes the budget, a 1.5x predicted
trace fails it (~$22.4k vs $20k cap, exit 1) -- the regression caught
without re-running the agent. 39 pytest cases; Go suite unchanged.
Every SPEC milestone and stretch is now implemented.
The gate decides on p95, and the gaussian baseline understates that tail two ways. Both fixes are opt-in; gaussian / independent stays the default so no unrequested assumptions sneak in. - fit --dist quantile: fit the conditional quantiles directly by quantile regression (exact LP via scipy.optimize.linprog/HiGHS) instead of an OLS mean + symmetric band. Targets the p95 the gate uses and represents the right-skew; crossing quantiles repaired by monotone rearrangement. Falls back to the empirical marginal quantiles when too few points -- still skewed, and flagged. report shows the p50/p95 lines and pseudo-R1; predict prints p50 and p95. - emit-trace --run-correlation R: a Gaussian copula shares one verbosity draw across a run's calls (z = sqrt(R)*z_run + sqrt(1-R)*z_idio). Without it, per-call independence understates the per-run total variance -- which is exactly what the gate's p95 is taken over. R=0 (default) is the prior independent behaviour. The same z drives both modes: gaussian uses it as the standardised residual, quantile maps z->u=Phi(z) and inverts the CDF, preserving skew. Also fixes a real portability bug: report/predict printed non-ASCII (arrow, +-, middot, en-dash) that crashed on a cp1252 Windows console -- output is now ASCII. Verified against a heavy-tailed recorded ground truth (p95 ~$0.0130/req): the quantile projection ($0.0126, CI [$0.0117,$0.0138]) covers it; the gaussian projection ($0.0121, CI [$0.0115,$0.0124]) sits lower with a falsely tight CI that misses it. 57 pytest cases (was 39); Go suite unchanged. New dep: scipy (quantile LP).
First dogfood against a real agent (LangGraph supervisor multi-agent + tools + guardrails), not a synthetic trace. examples/cloudoracle/. Capture path is the SPEC's documented proxy fallback (ADR D1): the agent talks to its model natively through LangChain, so instead of the OpenAI-compatible proxy we attach a LangChain callback to the one chat model the whole graph shares and write Augur's exact trace.jsonl schema from the usage every call reports. augur_dogfood.py imports the agent's public pieces (build_supervisor_graph/build_tools/run_guarded) and builds them with our model -- CloudOracle's source is never edited. Supports --provider anthropic (default; Claude's higher limits make a clean run practical) or gemini (--rps throttles the free tier). Result (Claude Haiku 4.5, 20 runs): gate PASS at $/request p95 $0.0198 (budget $0.02). The find-savings scenario is the headline -- its p95 cost is 2.3x its median, driven by a call-count tail (5 -> 13 calls/run when the savings specialist's ReAct loop keeps going). That agentic cost driver is exactly what Augur exists to catch, observed on a real agent; gating on the mean would hide it. Findings the dogfood surfaced (documented in the example README): - the agent isn't provider-portable: its LLM judge sends a system-only message Gemini tolerates but Anthropic rejects (--no-judge works around it; the fix belongs in CloudOracle). - Claude Haiku hallucinates dollar figures, tripping the agent's grounding fallback -- a quality signal orthogonal to cost. - the sidecar's output model is weak here (R1~0.03): output length is driven by call role, not input length -- reported honestly, motivating the documented next step (segment by role/seq). Includes scenarios/traffic/budget and a Gemini price snapshot. The Claude run prices against the default pricing.yaml, which already lists Claude.
- README: new "Dogfooded on a real agent" section — the callback-shim integration, the gate PASS, the find-savings call-count tail (p95 2.3x median), and the findings, linking the CloudOracle repo and the example. - SPEC: mark open question #1 (first dogfood target) resolved, done on CloudOracle's Insights Agent. - examples/cloudoracle/README: real CloudOracle repo link (github.com/Cro22/CloudOracle) and the sibling-project framing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.