September release candidate (DO NOT MERGE) - #1112
Draft
ChristianPavilonis wants to merge 380 commits into
Draft
Conversation
Resolve conflicts from the EdgeZero PR #257 sync on main (#761): - Cargo.toml: adopt main's crate renames, fastly/log-fastly 0.12, and edgezero tracking the upstream main branch; keep the branch's glob dep. - publisher.rs: keep both sides' new tests; forward-port test body extraction to the Option-returning Body::into_bytes API and drop the now-unused response_body_string helper superseded by the branch tests. - auction/endpoints.rs: unwrap_or_default the Option-returning into_bytes. - Relocate the new GPT SPA tests into the renamed crates/trusted-server-js tree and refresh stale crates/js doc-comment paths. - Take main's CI-validated integration-tests lockfile (deps unchanged on the branch).
The EdgeZero sync (#761) renamed crates/js and crates/integration-tests to crates/trusted-server-*. The old directories still hold local-only build artifacts (node_modules, target, dist) whose gitignore rules moved with the rename, so git now sees them as untracked. Ignore the defunct paths until the directories are removed from disk.
P1 — EdgeZero finalize cache/Set-Cookie privacy parity: Share the protected finalizer between the legacy and EdgeZero paths. apply_finalize_headers now strips surrogate cache headers and downgrades cookie-bearing responses to private, and skips operator response_headers that would re-enable shared caching on uncacheable responses; finalize_response delegates to it. The EdgeZero entry point re-applies an HttpResponse enforce_set_cookie_cache_privacy after ec_finalize_response and request-filter effects so a late EC Set-Cookie cannot reach a shared cache. Adds middleware tests for both cases. P1 — empty page-bids must not enable GPT services: adInit() only enables GPT services when it has a slot to display or refresh, and the SPA hook skips adInit() for an empty page-bids response unless prior TS state needs sweeping. Prevents a consent-denied or kill-switched navigation from activating the publisher's GPT setup. P2 — scope Prebid refresh targeting to the refreshed slots: setTargetingForGPTAsync is called with the synthetic refresh ad-unit codes so a one-slot refresh no longer mutates unrelated GPT slots. P2 — validate nested slot value shapes at build time: The creative-slot build check now validates media_type against the runtime MediaType variants, targeting as a string map, page_patterns as strings, providers.aps.slot_id as a string, providers.prebid.bidders as a map, and floor_price as a number — closing build-green/runtime-broken gaps. A drift-guard test ties media_type to the runtime enum. CI — suppress CodeQL cleartext-logging false positives: Annotate the provider/mediator "not registered" warnings; they log static config identifiers, not secrets.
The merge took main's trusted-server-integration-tests Cargo.lock, but the branch's trusted-server-core now pulls in glob (the creative-slot build check uses glob::Pattern). The integration crate path-depends on core, so its locked graph was missing glob and the --locked CI build refused to update it. Add only glob v0.3.3; no other versions change, keeping the shared direct-dependency parity check green.
…tes-impl Reconcile the server-side ad-template/auction work with main's EdgeZero canary rollout, the Axum/Cloudflare/Spin adapters, and the edgezero into_bytes() repin. - main(): route to EdgeZero only when the canary rollout selects it and the settings carry no creative_opportunity slots (combine both gates). - Thread the new handle_publisher_request(kv, ec_context, auction) parameters through the Axum, Cloudflare, and Spin adapters with an empty AuctionDispatch — server-side auction stays deferred there, as on the Fastly EdgeZero path. - Keep HEAD's apply_floor_prices None-price drop and the corrected publisher OwnedProcessResponseParams / single stream_publisher_body; union the diverged publisher, html_processor, and orchestrator tests. - Drop the stale into_bytes().unwrap_or_default() test calls and add the new HtmlProcessorConfig / PlatformBackendSpec fields and the route_request slots argument to the bench and adapter tests.
Blocking: - Move tokio to [dev-dependencies] in trusted-server-core; it was only used by #[tokio::test] and was linking the runtime into the wasm prod build. Confirmed the release wasm adapter build no longer pulls tokio. - Roll back the SPA currentPath on a failed /__ts/page-bids fetch so a transient error no longer permanently strands that route (gpt/index.ts). Build/runtime parity and diagnostics: - Bound creative-opportunity format width/height to u32 range at build time so values the runtime u32 cannot hold are rejected early. - Add #[serde(deny_unknown_fields)] to the build.rs config stub to match the runtime type and reject mistyped table keys at build time. - Warn when the <body> end-tag handler is absent so a silently non-rendering server-side ad feature is diagnosable. - Log dropped slot bidders that are neither configured nor the aps provider. - Log build_bid_index collisions (multiple bids per seat/imp). JS correctness: - Narrow uid.atype to a number before the range check in sanitizeAuctionUid. - Resolve findInjectedSlotForRefresh by exact/container match before the prefix fallback, with a regression test for prefix-overlapping div_ids. - Guard the gpt_bootstrap prefix scan against an empty div_id. - Route injectAdmIntoSlot through findSlotElementByDivId for consistency. Cleanup and docs: - Remove the dead has_post_processors routing dependency from classify_response_route and (now unused) handle_publisher_request. - Extract the duplicated EID resolution/consent-gating/device tail shared by the initial-page and page-bids dispatch paths into one helper. - Anchor the surrogate cache-header list in a shared const so the legacy and EdgeZero Set-Cookie privacy paths stay aligned. - Refresh stale docs (PublisherResponse::Stream, the publisher module platform-coupling note, and UserInfo.eids consent-gate location).
Moving tokio to trusted-server-core dev-dependencies removed it from the crate's normal dependency list, so the integration-tests lockfile (which resolves core's non-dev deps) no longer pins tokio under core. Keeps `cargo --locked` green for the integration job.
These EdgeZero-style adapters finalize buffered, and the sync
`buffer_publisher_response` drives `stream_publisher_body`, which ignores
`params.dispatched_auction` — so they injected an empty `tsjs.bids = {}`
while Fastly (legacy streaming finalize) served real bids.
- Add `buffer_publisher_response_async` in core: for the Stream variant it
drives `stream_publisher_body_async`, which awaits
`collect_dispatched_auction`, writes `ad_bids_state`, and injects the bids
before `</body>`.
- Pass the configured `creative_opportunities.slot` (not empty) to
`handle_publisher_request` on all three adapters; it matches them against
the request path internally.
- Call the async finalize from each adapter (Cloudflare/Spin via their
now-async `resolve_publisher_response`).
EID targeting stays off for now (these adapters pass `kv: None`).
…ters The auction consent gate (`consent_allows_server_side_auction`) reads jurisdiction and TCF consent from the EC context. The adapters passed `EcContext::default()` to `handle_publisher_request`, leaving jurisdiction Unknown with no consent — so the gate failed closed and no auction ran (empty `tsjs.bids`), even though the slots matched. Build the context via `read_from_request_with_geo` (consent from the request, geo from the platform), mirroring the Fastly entry point, and fall back to default on a parse error. Cloudflare resolves geo from the Workers `cf` object when deployed; Axum and Spin have no-op geo providers, so on those a known non-GDPR jurisdiction requires the request to carry geo or the gate needs a TCF consent signal.
When the auction does not run, this pinpoints which gate suppressed it (slots, bot, navigation, consent, or orchestrator kill switch) instead of only seeing `dispatch_auction: None`. Pair with the EC-context jurisdiction log when consent_allows_auction is false.
The EdgeZero buffered path passed empty slots and finalized via the sync `buffer_publisher_response`, so configured creative-opportunity slots were routed to the legacy path by `edgezero_can_handle_settings`. Now that `buffer_publisher_response_async` collects the dispatched auction, EdgeZero can run the full ad stack: - Pass the configured `creative_opportunities.slot` and finalize via `buffer_publisher_response_async` (the path's `ec.ec_context` already carries consent + platform geo). EID targeting stays off (`registry: None`). - Drop the `edgezero_can_handle_settings` gate, its routing branch, the three tests, and the now-unused test settings helpers — EdgeZero handles configured slots, so the legacy fallback for them is obsolete.
The EdgeZero publisher path dispatched the auction with registry: None, so the bid request carried no KV identity-graph EIDs (only client cookie EIDs). It already passes ec.kv_graph as the identity KV, so wire the matching PartnerRegistry::from_config(settings.ec.partners) into the AuctionDispatch to resolve server-side partner EIDs — matching the legacy auction path. Fastly-only: the sync EC identity graph (KvIdentityGraph/EcKvStore) works on Fastly's sync KV; the async-KV portability adapters are unaffected (they still pass registry: None until the EC graph supports async stores).
Resolve the fifth code-review pass. The blocking findings were all cross-adapter parity gaps in the server-side auction: - Build the geo-aware EC context in the /auction handlers on Axum, Cloudflare, and Spin. They passed EcContext::default(), leaving jurisdiction Unknown and failing the consent gate closed even for consented users. A shared per-adapter build_ec_context helper now serves /auction, page-bids, and the publisher fallback, and logs (rather than swallows) a malformed-consent read error. - Wire GET /__ts/page-bids and its OPTIONS->403 CSRF guard on the Fastly EdgeZero path and all three portability adapters, reusing core handle_page_bids and a shared page_bids_preflight_denied() helper. Previously it was Fastly-legacy-only, so SPA re-auction silently fell through to the origin on every other path. - Add trusted_server_core::response_privacy with the Set-Cookie cache-privacy downgrade and the uncacheable-operator-header guard, and call it from every adapter's apply_finalize_headers so a shared cache (Cloudflare) can no longer serve an operator/origin public Cache-Control on a cookie-bearing response. Also address the inline and non-blocking findings: warn on a dropped dispatched auction for bodiless responses, extract build_slot_json shared by the initial-page and page-bids paths, use creative_opportunity_slots() everywhere, drop the PBS id->ad_id fallback, log APS slot-id collisions, align the parallel provider parse with the collect path, remove the dead sync buffer_publisher_response, factor the mediator placeholder request, drop the unused toml dependency, guard MediaType against a future serde(default), and document the Fastly-only KV EID enrichment. JS: dedup win/billing beacons across concurrent renders, add the SSR guard to installSlimPrebidLoader, and short-circuit waitForSlotElements on an already-aborted signal. Add regression tests for the currentPath rollback and the u32::MAX format-dimension rejection.
Adopt main's ts-CLI config model over the branch's build-time embedding: drop the build.rs config generation and [build-dependencies], accept the trusted-server.toml deletion, remove the now-orphaned creative_slot_build_check module, and migrate the [creative_opportunities] example into trusted-server.example.toml. Slot validation is preserved via Settings::prepare_runtime, which the CLI runs at config push time. Adopt main's refactored EdgeZero finalize path in the Fastly adapter and centralize the per-user Set-Cookie cache-privacy guard in send_edgezero_response so it covers every EdgeZero send path. Fix three core test helpers for main's Body::into_bytes -> Option change.
Rebuilds feature/ts-cli-ad-templates fresh on server-side-ad-templates-impl (which already carries the ts CLI #799 + audit #800 via main), dropping the redundant ts-cli-base history that caused the merge conflicts. Adds the ad-template CLI: - ts audit ad-templates verify — browser/CDP ad-template slot verification - ts audit page — read-only page summary - ts config ad-templates — server-side ad-template diagnostics - shared CLI app-config loader + ad-template evidence/output models Keeps #800's URL->draft-config bootstrap by relocating it under audit/generate/ and exposing it as ts audit generate. Core: extracts the server-side ad-stack gate into creative_opportunities::evaluate_ad_stack_gate (three-state Yes/No/Unknown) so the CLI and the runtime publisher path share one gate definition.
- Stop forwarding client-supplied X-Forwarded-For to Prebid Server; synthesize it from the platform-attested client IP instead - Make the APS slot ID remapping request-scoped by threading the AuctionRequest through parse_response_with_context, removing the shared provider-instance mutex that concurrent auctions could race - Guard dispatch_auction against multi-provider fan-out on platforms whose HTTP client executes requests sequentially, mirroring run_providers_parallel - Reject negative and non-finite creative-opportunity floor prices at config validation - Re-run the Set-Cookie cache-privacy downgrade after operator response headers are applied so configured Set-Cookie plus public cache headers cannot produce a shared-cacheable response - Install Prebid user ID modules immediately when the bundle loads after window.load (slim-Prebid lazy path), with a once-only listener - Drop allow-same-origin from debug ADM fallback iframes and validate extracted iframe src schemes to http(s) only
Reconstruct [creative_opportunities] slots from a live page's GPT registry and gampad/ads requests, and write them into an existing trusted-server.toml in place, preserving all other sections. Slots merge across runs: --page-pattern unions patterns into a re-seen slot, existing slots are preserved, and --replace wipes. Ephemeral div-id noise (React hashes, -container, hex UUIDs) is normalized to stable prefixes so verify matches across renders, and TOML keys/strings are escaped defensively. Add --cookie to ad-templates generate and verify so a valid bot-protection clearance cookie can carry the browser audit past an origin challenge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile the ad-template CLI + generate feature onto the commands/ module layout the base branch adopted from main (ts dev proxy, #798): - audit/ -> commands/audit/ : mod.rs is the audit namespace; the #800 draft-gen plus slot reconstruction move under commands/audit/generate/ (base's duplicate commands/audit/{analyzer,browser_collector} dropped). - config_ad_templates.rs -> commands/config/ad_templates.rs. - app_config and ad_templates support modules stay at crate root. - run.rs/lib.rs wiring merged with the new `ts dev` command. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Roll SPA page-bids navigation `currentPath` back to the last applied path instead of the immediately-previous one, so an aborted-then-failed navigation can no longer strand a route behind the no-op guard; add a regression test. - Add a concurrent render-bridge test: two same-adId messages before the cache fetch resolves must collapse to one fetch (in-flight gate), two beacons. - Assert `OPTIONS /__ts/page-bids` is denied with 403 on every adapter (Axum/Cloudflare/Spin) in cross-adapter parity. - Add a `u32::MAX` banner-format test covering the imp-drop branch when all formats exceed `i32::MAX`. - Dedup the page-bids GET 403 into `page_bids_preflight_denied()`. - Fix stale comments/docs: `buffer_publisher_response_async`, soften the oversized-body comment, and correct the `firedBeacons` key doc.
Resolve #744 (legacy entry-point removal + streaming publisher) against the server-side auction: keep the buffered auction path on all adapters, supersede the streaming publisher for publisher navigation. Verified: fastly/core/spin/ axum/cloudflare tests + clippy + fmt clean.
- Escape page-controlled slot fields and validate the gampad network id before splicing scraped values into trusted-server.toml - Add navigation and teardown timeouts to the verify browser collector - Snapshot evidence before the scroll pass so load-time entries keep phase initial_load; add a Chrome-gated regression fixture - Cap collector evidence lists in the injected script and after decode - Preserve CRLF line endings and render non-finite floor_price as valid TOML when updating configs in place - Cover all 128 gate combinations in the core ad-stack mirror test - Extract slot TOML rendering/merging/splicing into slot_toml.rs
…nto feature/ts-cli-ad-templates
- Recognize [creative_opportunities] table headers carrying inline comments in the in-place splice and replace_key_in_section, so a valid operator config is updated instead of gaining a duplicate section - Default generated page_patterns from the recorded post-redirect final URL instead of the requested URL, falling back to the requested URL when the recorded final URL is invalid - Escape DEL (U+007F) in toml_string, which TOML basic strings reject alongside chars below U+0020 Each fix carries a parse-backed regression test.
Main squash-merged the same server-side ad-templates feature (#680), then added Tinybird auction telemetry (#818), edgezero v0.0.4 pins (#862), and external first-party Prebid bundle loading (#743). Resolutions: - Files where this branch matched the impl branch tip take main's version (telemetry/prebid-bundle additions on top of identical feature code): orchestrator, endpoints, openrtb, prebid integration, fastly app, .gitignore, prebid docs - creative_opportunities.rs keeps this branch's version (adds the shared ad-stack gate and pattern machinery used by the ts CLI; main's copy was identical minus those additions) - publisher.rs takes main's version with this branch's should_run_server_side_ad_stack delegation to evaluate_ad_stack_gate re-applied - run.rs keeps both sides' CLI tests (ad-templates/audit + prebid bundle)
The access emitter carried the configured body limit without enforcing it, allowing oversized rows to bypass the intended transport safeguard.
…erver into edgezero-secrets
An APS bid that wins Prebid targeting is served by Ad Manager as a 1x1 universal creative that resizes itself only after the creative draws. Every guard on that render path returned silently, so a slot that never drew was indistinguishable from one that did: Ad Manager reports a non-empty 1x1 render either way, and the tester framework reports "filled". Name the guard that stopped the render instead. The sandboxed renderer document now reports bad_hash, source_mismatch, nonce_mismatch, descriptor_keys, descriptor_fields, descriptor_envelope, and amazon_script_error on the existing failure message. Reporting is one-shot and answers through the parent, never the sender, so an unrelated sender cannot consume the frame's single report or learn anything from it. Traffic that is not shaped like the render handshake stays silent as before. The Universal Creative source labels its own frame_timeout and frame_load_error, and relays whichever reason it holds to the top window. That relay crosses an origin boundary, so reasons resolve through a null-prototype allowlist that drops anything unlisted and leaves a hostile __proto__ or constructor as undefined. Reasons are fixed categories. A descriptor is never echoed back.
The APS capability handshake never told diagnostics anything, so every request cycle on that path reported `delivery: unknown` and no creative failures at all. On a live page that meant 24 of 24 cycles were unattributed while APS bids were winning and rendering blank, which is the state that made this hard to diagnose from the outside. Record the attempt around the handshake. The path runs on the publisher's own Prebid ad units, which never pass through Trusted Server slot mapping, so no creative opportunity exists for them and the store would reject the attempt as `creative_request_without_slot`. Resolve the GPT slot by element ID and record the opportunity first. Each silent return that ends in a blank now names itself: aps_consumed_tombstone, aps_source_not_in_ad_unit, aps_descriptor_fields, aps_tombstone_capacity, and aps_missing_renderer_url. A successful post records a response. Consumed ad IDs carry the attempt they were served under, so a replay, or a failure the creative frame relays after the fact, is attributed to the render it belongs to rather than guessed at. The relay listener treats the creative as untrusted: the reason must resolve through the allowlist, the attempt comes from our own tombstone rather than the message, and it never answers the sender.
# Conflicts: # crates/trusted-server-adapter-axum/src/app.rs # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-core/src/proxy.rs
# Conflicts: # Cargo.lock # crates/trusted-server-adapter-spin/src/platform.rs # crates/trusted-server-core/src/auction_config_types.rs # crates/trusted-server-core/src/config.rs # crates/trusted-server-core/src/config_payload.rs # crates/trusted-server-core/src/integrations/prebid.rs # crates/trusted-server-core/src/settings.rs # docs/guide/api-reference.md # docs/guide/configuration.md # docs/guide/ec-setup-guide.md # docs/guide/error-reference.md # docs/guide/fastly.md # docs/guide/first-party-proxy.md # docs/guide/getting-started.md # docs/guide/proxy-signing.md # scripts/template-cache-local-test.sh # trusted-server.example.toml
# Conflicts: # Cargo.lock
# Conflicts: # crates/trusted-server-cli/src/commands/audit/mod.rs # crates/trusted-server-core/src/config.rs # docs/guide/getting-started.md
# Conflicts: # crates/trusted-server-core/src/platform/test_support.rs # crates/trusted-server-core/src/publisher.rs
# Conflicts: # crates/trusted-server-adapter-fastly/src/app.rs # crates/trusted-server-adapter-fastly/src/main.rs # crates/trusted-server-adapter-fastly/src/tinybird.rs # crates/trusted-server-core/src/publisher.rs # crates/trusted-server-core/src/settings.rs # trusted-server.example.toml
Adds section 18 to the request phase timing spec: three first-call-wins T0 offsets (auction dispatched, resolved, committed) on RequestTimings, emitted as additive nullable columns on access_logs_raw with auction_id as the join key to the per-bidder auction dataset. Answers the overlap-proof questions the two existing clocks cannot: when the auction started relative to request entry, when the final bid landed, and when targeting was committed toward GAM.
Implements spec section 18: three first-call-wins marks on RequestTimings (dispatched at the DispatchAuctionOutcome::Dispatched arm, resolved after collect at both sites, committed after write_bids_to_state at both sites), carried through TimingSnapshot into four additive access_logs_raw columns: auction_dispatched_ms, auction_resolved_ms, auction_committed_ms, and auction_id as the join key to the per-bidder auction dataset. Null offsets mean no auction ran; a failed dispatch records nothing. FORWARD_QUERY fills the new columns with typed defaults for pre-existing rows. No header emission, no config surface, no adapter changes: the values ride the existing snapshot and the tinybird.access_enabled gate.
The Cloudflare integration harness writes wrangler.integration.generated.toml at test time; it was swept into the previous commit by accident. Ignore it so local CI=1 runs cannot commit it again.
# Conflicts: # crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts
# Conflicts: # crates/trusted-server-js/lib/src/integrations/gpt/index.ts # crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts
# Conflicts: # crates/trusted-server-core/src/publisher.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
September release candidate
This PR is an integration branch for release-candidate validation. Do not merge it into
main.Already included through main:
Included PRs