test(dwctl): golden equivalence transcripts for the inference pipeline - #1502
Open
sejori wants to merge 2 commits into
Open
test(dwctl): golden equivalence transcripts for the inference pipeline#1502sejori wants to merge 2 commits into
sejori wants to merge 2 commits into
Conversation
…eline Snapshot the full observable surface of one request through the /ai/v1 stack - client response, raw upstream-bound body + control headers, outlet http_requests/http_responses rows, and the fusillade row - for 10 corpus cases across chat completions, responses (incl. background), Anthropic /messages, legacy /completions, and embeddings. These are refactor safety-nets ahead of the parse-once pipeline change: any byte-level drift in what we forward, log, or store fails CI and must land as a reviewed snapshot diff. Two currently-untested facts are locked deliberately: client-supplied id/completion_id fields are forwarded upstream on the realtime path (only the stored copy is scrubbed), and legacy /completions bypasses the inference middleware while still receiving stream-flag injection.
Deploying control-layer with
|
| Latest commit: |
2b72aff
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7a88eb5d.control-layer.pages.dev |
| Branch Preview URL: | https://test-golden-inference-transc.control-layer.pages.dev |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an insta-based golden snapshot corpus that captures end-to-end “equivalence transcripts” for a single inference request through the full /ai/v1 pipeline (client response, upstream-bound request bytes, outlet persistence, and fusillade persistence) across a set of representative request shapes.
Changes:
- Introduces
dwctl::test::golden_transcriptswith integration-style tests that provision models/endpoints via admin APIs, drive/ai/v1/*requests, and snapshot a normalized transcript. - Adds 10 new
.snapgolden transcript fixtures covering chat completions, responses (blocking/streaming/background), Anthropic/messages(blocking/streaming), legacy/completionsstreaming, and embeddings. - Adds new dev-dependencies (
insta,regex) to support snapshot testing and transcript normalization.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| dwctl/src/test/golden_transcripts.rs | New golden transcript test harness and corpus cases for /ai/v1 inference pipeline equivalence. |
| dwctl/src/test/mod.rs | Registers the new golden_transcripts test module. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_responses_streaming.snap | New golden transcript snapshot for streaming /responses. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_responses_blocking.snap | New golden transcript snapshot for blocking /responses. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_responses_background.snap | New golden transcript snapshot for background /responses (202 + poll). |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_legacy_completions_streaming.snap | New golden transcript snapshot for legacy /completions streaming behavior and no-intercept invariant. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_embeddings_blocking.snap | New golden transcript snapshot for blocking /embeddings. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_chat_completions_streaming.snap | New golden transcript snapshot for streaming chat completions. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_chat_completions_blocking.snap | New golden transcript snapshot for blocking chat completions. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_chat_client_supplied_ids.snap | New golden transcript snapshot locking client-supplied id forwarding behavior. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_anthropic_messages_streaming.snap | New golden transcript snapshot for streaming Anthropic-compatible /messages. |
| dwctl/src/test/snapshots/dwctl__test__golden_transcripts__golden_anthropic_messages_blocking.snap | New golden transcript snapshot for blocking Anthropic-compatible /messages. |
| dwctl/Cargo.toml | Adds insta and regex dev-dependencies for the golden transcript tests. |
| Cargo.lock | Locks new transitive dependencies introduced by insta/regex. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+256
to
+260
| let query = | ||
| format!("SELECT to_jsonb(t)::text FROM {table} t WHERE to_jsonb(t)::text LIKE '%{needle}%' ORDER BY to_jsonb(t)->>'id' LIMIT 1"); | ||
| for _ in 0..200 { | ||
| let row: Option<(String,)> = sqlx::query_as(&query).fetch_optional(pool).await.expect("poll row"); | ||
| if let Some((json,)) = row { |
Comment on lines
+312
to
+315
| let query = "SELECT r.state, r.service_tier, t.model, t.method, t.path, t.endpoint, t.body \ | ||
| FROM fusillade.requests r JOIN fusillade.request_templates t ON t.id = r.template_id \ | ||
| WHERE t.body LIKE $1 AND r.state IN ('completed', 'failed', 'cancelled') \ | ||
| LIMIT 1"; |
Comment on lines
+192
to
+211
| // ─── Normalization ──────────────────────────────────────────────────────────── | ||
|
|
||
| /// Scrub run-to-run volatility out of a transcript chunk so snapshots are | ||
| /// stable: UUIDs, ephemeral ports, epoch + ISO timestamps, and the per-run API | ||
| /// key secret. Deliberately does NOT touch structure or key order — byte-level | ||
| /// differences in the bodies are exactly what these tests exist to catch. | ||
| fn normalize(text: &str, api_key: &str) -> String { | ||
| let uuid = regex::Regex::new(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}").unwrap(); | ||
| let port = regex::Regex::new(r"127\.0\.0\.1:\d+").unwrap(); | ||
| let iso_ts = regex::Regex::new(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(\.\d+)?([+-]\d{2}:?\d{2}|Z)?").unwrap(); | ||
| let epoch_fields = | ||
| regex::Regex::new(r#""(created|created_at|completed_at|failed_at|cancelled_at|timestamp|expires_at)"\s*:\s*\d{9,}"#).unwrap(); | ||
|
|
||
| let text = text.replace(api_key, "<api-key>"); | ||
| let text = uuid.replace_all(&text, "<uuid>"); | ||
| let text = port.replace_all(&text, "127.0.0.1:<port>"); | ||
| let text = iso_ts.replace_all(&text, "<ts>"); | ||
| let text = epoch_fields.replace_all(&text, r#""$1":0"#).to_string(); | ||
| text | ||
| } |
…cripts
CI (nextest, process-per-test) starts the Responses translator's global
output-item counter at zero for every test, while local cargo test shares
one process across tests - so item_{:016x} ids differed between the two
runners. Normalize them to item_<n>; the counter's absolute value carries
no meaning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a golden-snapshot corpus (insta) that locks the full observable surface of one inference request through the /ai/v1 stack, per case:
10 cases: chat completions (blocking, streaming, client-supplied ids), responses (blocking, streaming, background 202→poll), Anthropic /messages (blocking, streaming), legacy /completions streaming, embeddings. Volatile fields (uuids, ports, timestamps, key secrets) are normalized; determinism verified across three consecutive runs.
Why
Safety net ahead of the parse-once pipeline refactor (params struct after inference_middleware; shared canonical body after translation; single serialize in outbound_request). Existing e2e tests are behavior specs asserting a handful of named fields — a refactor can change upstream bytes, logged bodies, and stored rows without failing any of them. These transcripts fail on any drift, so equivalence is proven against recorded truth and intentional changes land as reviewed snapshot diffs.
Two currently-untested facts are locked deliberately, both slated to change in the follow-up PRs:
Updating snapshots
Run the golden tests with INSTA_UPDATE=always and review the .snap diffs like any other code change.
Out of scope (noted in the module doc)
Flex/background daemon dispatch, the prompt-cache layer, tool injection, non-strict onwards mode.