Conversation
Mirrors app/agent/usage.py in the evoflux repo so a Conductor-computed price and an EvoFlux-computed price agree whenever they price the same call from the same models.dev catalog data: cache-read and cache-write tokens are billed at their own rate and subtracted from the plain input count before it is billed, the same for reasoning tokens against output, and a request past a long-context threshold picks up the highest applicable tier. Also adds cache_savings_usd, the net-USD-saved counterpart used by later commits to report how much prompt caching is actually worth. Pure and I/O-free — catalog fetch/caching lives in conductor-server.
Mirrors the evoflux client's own catalog client (app/agent/providers/model_registry.py): fetches and caches https://models.dev/api.json independently with a 24h TTL disk cache, so Conductor can price a model_call on its own when the reporting client did not send a cost. Includes the evoflux provider-id alias table (googlegenai -> google, bedrock -> amazon-bedrock, etc, ported verbatim from app/agent/providers/registry.py) so a call reported under evoflux's own provider id still resolves to the models.dev row, and a zero-token special case so a call that consumed nothing prices at $0 instead of sitting "unpriced" forever because its provider/model has no catalog rate. AppState gains a pricing seam (ModelPricingProvider) the same shape as the existing host_metrics one, so HTTP tests can inject a fixed StaticPricingCatalog instead of reaching the network.
… ingest Adds cache_write_tokens next to the existing cache_read_tokens throughout the telemetry pipeline (wire schema, storage, request-detail response), so a model_call's cache-write activity is stored and returned instead of silently dropped. Also backfills cost server-side for a model_call event whose reporting client did not price it: ingest now calls into the models.dev catalog (core::model_pricing) for any unpriced event, tags the result conductor_catalog, and never overwrites a cost the client already computed. A one-off backfill_model_pricing binary reprices historical rows the same way, for calls that predate this or that no catalog rate existed for at the time. Reserves cache_savings_usd_micros on the wire and in storage (nullable, signed — it can be negative) for a later commit to compute; this commit only carries the plumbing so that field has somewhere to live.
Exposes the models.dev catalog conductor-server already fetches for cost backfill (core::model_pricing) as its own read endpoint — GET /model-pricing lists every priced model with its resolved rates, POST /model-pricing/refresh forces a re-fetch — for a reference-price screen, distinct from the usage/spend reporting under /analytics/resource-usage. The world AI pricing catalog is public reference data, not scoped to any project or organization, so the new model_pricing.read permission is granted to every role. Route/permission/action counts and the generated route-inventory snapshot are updated to match.
New /app/resources/pricing screen: search, provider/recommendation filters via a searchable combobox (shared/ui/searchable-select, wrapping @base-ui/react's Combobox), sortable columns and pagination over the world AI pricing catalog the server now exposes, with a manual refresh action. Nav entry and route are gated on the new model_pricing.read permission, which every role holds.
TelemetryEventDetail.cost_source only typed "evoflux_catalog", so a conductor-priced event (see the ingest backfill) would have rendered its cost-source label as undefined. Widen the type and give it a label.
The /analytics/resource-usage endpoint has always priced, counted and listed everything under one of two scopes, but only the Dashboard could switch between them — the Usage page was governed-only, so "how much of our EvoFlux activity isn't attributed to a managed resource at all" had no answer there. Relocates the toggle out of dashboard/ into resource-usage/ as ActivityScopeTabs (generic over any screen built on that endpoint) and wires it into the Usage page's Overview tab, alongside the state/URL plumbing (requestedScope, scope-locked pages force governed since resource_kind filtering only works in the governed join path). Also carries this session's Usage-page overview additions for the period-comparison work landing in a later commit (a 5th stat card and its delta badge) — they touch the same render block as the scope tabs and aren't worth re-splitting out.
These figures are real calculations against the models.dev catalog, not vague estimates — "Cost" says what they are without implying otherwise. Purely a label change; StatCard.hint also widens from string to ReactNode here since a later commit needs to render a delta badge inside it, not just text.
Ports the visual/math approach from evoflux's own waterfall chart (web/src/routes/telemetry/waterfall) to conductor's request-detail pages: bars positioned from each event's reported_at minus duration_ms, colored by event_type/status, clicking one highlights its row below. request-event-timeline.tsx combines the waterfall with the existing event list — now shared by both the Usage and Member request-detail pages, which previously each hand-rolled their own <ol> — and rounds out every event's token grid to all 6 categories (input, output, cache read, cache write, reasoning, tool use); the member page's version was missing tool_use entirely.
The custom date-range filter (used across 5 pages via useUsageRange/ DateRangeFilter) fell back to two native <input type="date">, which looks and behaves inconsistently across browsers and doesn't support picking a range in one control. Adds shadcn-style Calendar (react-day-picker) and Popover (@base-ui/ react, matching every other overlay primitive in shared/ui/ — not radix-ui) primitives, and rewrites DateRangeFilter around them. Range selection uses a draft state committed only on Apply: react-day-picker sets `to = from` on the very first click when no `min` is given (addToRange.js), so treating "`to` is set" as "the user is done" closed the popover before a second click was possible.
…rison Sums cache_write_tokens and the new cache_savings_usd_micros (net USD saved by cache discounts, minus the cache-write premium — see conductor_domain::pricing::cache_savings_usd) into ResourceUsageTotals and ResourceUsageModel, alongside a backfill_cache_savings binary for the 100% of historical rows the new column starts NULL for. Also adds an opt-in comparison: ResourceUsageQuery::compare_previous, when set, has analytics() price the equal-length window immediately before the one being viewed (previous_from = from - (to - from)) through a new leaner period_totals() query, run concurrently with the existing panels rather than as an extra round trip. Exposed as ResourceUsageAnalytics.previous_period and a compare_previous query param on GET /analytics/resource-usage — no new route or permission, so the route manifest is untouched.
PeriodDeltaBadge renders a small colored delta ("+$0.30 vs previous
period") comparing a current figure against ResourceUsageAnalytics.
previous_period, green when the change is favorable and amber when it
isn't (which direction is favorable is a prop — lower cost is good,
higher savings is good).
Wires it into: a new "Cache savings" strip under the Dashboard's metric
grid and a matching stat card on the Usage page's Overview tab (both
next to a delta badge on Cost too), and a Cache/savings column on the
per-model breakdown table. Both pages now request compare_previous.
Also carries the ModelPricing* response types the earlier Model Pricing
Catalog commit needed but landed in the same source hunk as
ResourceUsagePeriodTotals, and the model_pricing.read permission key
that commit's nav gate and route also needed.
…hanges The Dashboard/loading-state suite was fully broken by the manifest self-check added for model_pricing.read: the fixture's mocked authorization payload didn't list it, so every page failed closed with "incomplete permission metadata" before rendering anything. Adds it to the fixture's permission set (all roles, matching the real grant). Also updates two assertions left pointing at the old "Estimated cost" label and the old "Dashboard activity scope" group name (now generic "Activity scope", since the toggle is shared with the Usage page), and gives the dashboard fixture non-zero cache_write_tokens/ cache_savings_usd_micros/previous_period so the new UI has something real to show. Refreshes the 3 visual baselines the new Cache savings strip changes the layout of. All 15 e2e specs pass after these fixes.
6 tasks
Contributor
Author
|
Paired evoflux-side change: evoelsewhere/evoflux#20 (forwards |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Syncs Conductor's model-cost accounting with evoflux's own, and closes the UX gaps that surfaced while doing it:
app/agent/usage.py) into Rust, fetches the samemodels.devcatalog independently, and backfills cost server-side for calls the client left unpriced (provider-id aliases, a zero-token = $0 fix)./app/resources/pricingscreen exposing that catalog as reference rates (search, filter, sort).<input type="date">with a proper shadcn-style Calendar + Popover on@base-ui/reactprimitives.Companion evoflux-side change (forwards
cache_write_tokens, which this branch's schema now has somewhere to go): evoelsewhere/evoflux#20.Checklist
Correctness
registry.py(googlegenai→google,bedrock→amazon-bedrock, etc.) — these were previously silently unmatched.compare_previousadds no new route or permission — route-manifest tests (EXPECTED_ROUTE_ACTIONSetc.) are unaffected and untouched by that part of the diff.model_callevent regardless of who priced its cost (client or Conductor) — it's an independent derived figure, not part of the cost-authority chain.cache_savings_usd_microsstored signed/nullable (can be negative — a call that wrote far more to cache than it ever read back) and never clamped to zero in aggregate SQL.Compatibility / migration
cache_write_tokens,cache_savings_usd_micros) are additive, nullable-or-defaultedALTER TABLEs applied best-effort — safe against an existing DB.backfill_model_pricingandbackfill_cache_savingsone-off binaries bring historical rows in line; run against the dev DB as part of this PR's verification (see below).model_pricing.readpermission granted to every role (world AI pricing catalog is public reference data, not project-scoped).Tests
cargo test --workspace— see evidence below.npm run typecheck/npm run build(apps/web) — clean.apps/web,npm run test:e2e) — all specs pass; 3 visual baselines refreshed for the new Cache savings strip (screenshots below).model_pricing.readpermission broke the Dashboard/loading-state suite's fixture (client-side "incomplete permission metadata" fail-closed) — not a pre-existing failure, introduced by this branch and fixed within it."Estimated cost"→"Cost","Dashboard activity scope"→"Activity scope").Out of scope (called out explicitly, not silently skipped)
TokenCostChart(daily chart) — this PR's "saving" ask is an aggregate figure, not a new chart dimension.Test evidence
cargo test --workspace(Full log attached to this PR's CI run.)
Frontend
Playwright e2e (
apps/web,npm run test:e2e)Playwright screenshot evidence
Dashboard, populated admin view — "Cost" label, the new "Cache savings $0.94 ↑ $0.30 vs previous period" strip, Model Pricing nav entry all visible:
Same view at mobile width:
Loading state (structural shimmer, includes the new strip's skeleton row):
Live verification against the dev DB (
data/conductor.db)Manually recomputed one real row by hand (xiaomi/mimo-v2.5, rates $0.14 input / $0.0028 cache-read per 1M, 140,992 cache-read tokens) → 19,344 µUSD, exactly matching the stored
cache_savings_usd_micros.