feat(eve): benchmark loop runtime overhead

[ruiconti]

Description

Part of #512. This is stacked on #687, which is stacked on #513. The PR base is rui/loop-interface-prototypes.

This adds a correctness-gated benchmark for end-to-end eve agent latency and the runtime layers inside one turn. It compares inline execution, Workflow DevKit, and a real local Temporal server/Worker without changing the authoring API.

Runtime boundary

• Extract the existing session-create and turn-step behavior into runtime-neutral operations. Workflow, inline, and Temporal call the same code instead of maintaining benchmark copies.
• Keep backend-specific scheduling, persistence, retries, and park/rekey mechanics in each runtime adapter.
• Select benchmark runtimes only through an internal route boundary. Inline and Temporal fail closed in direct Vercel Functions because their state and worker are process-local.

Measurement contract

• Measure client E2E time on one monotonic clock, from POST acknowledgement through session.waiting, with the protocol phases and reducer work reported separately.
• Record neutral server intervals, engine attempt IDs, event observations, and park/rekey marks. Durations are calculated only inside one clock domain; the runner never subtracts client and server clocks.
• Write raw JSONL telemetry through an ordered, non-blocking process queue so filesystem completion is not part of the measured turn.
• Reject samples that do not produce the exact two-step transcript: one benchmark_echo tool call followed by the exact tool result.

Benchmark topology

• Run randomized complete blocks serially, reuse one nonce across the three runtimes in a block, discard warmups, and report per-runtime percentiles plus paired same-block differences.
• Default to a source-backed deterministic model so the comparison isolates the instrumented runtime stack. An explicit live lane uses openai/gpt-5.4 for provider-inclusive measurements.
• Run locally as three processes, or in one Vercel Sandbox pinned to an exact commit with one install/build and three detached processes. Provisioning and readiness are outside the samples.
• Authenticate Sandbox requests with Vercel OIDC. The token stays out of arguments, deterministic server environments, and JSONL records; only its project/environment claims configure the server-side verification boundary.

This is internal research tooling. It adds no public API and has no changeset.

How did you test your changes?

• Fixture suite: 14 files / 95 tests, fixture typecheck, runner compile, and fixture build.
• eve suite: 429 unit files / 4,439 passed / 1 skipped; 9 targeted telemetry integration tests; 1 Temporal runtime scenario; eve typecheck and JS build.
• Workspace checks: formatter, lint, invariant guard, frozen install, and diff whitespace check.
• Local deterministic smoke: one warmup and one measured block; all six runtime samples were valid and all server telemetry was complete.
• Vercel Sandbox smoke pinned to b183017cbd93821e035fa472eb9a5c884053ad36: run 421ceb71-479c-491b-9e96-2395cabf21b0 completed one warmup and one measured block with six valid samples and complete telemetry for inline, Workflow DevKit, and Temporal. This is an execution proof, not a comparative result; n=1 is too small to interpret.

PR Checklist

• I linked an issue with prior discussion confirming this change is wanted
• I ran the relevant checks from CONTRIBUTING.md
• I added tests and documentation where relevant
• I added a changeset if this touches the published eve package (not required: internal research tooling with no public behavior change)
• DCO sign-off passes for every commit (git commit --signoff)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
eve-docs	Ready	Preview, Comment, Open in v0	Jul 11, 2026 1:32am
eve-docs-1644	Ready	Preview, Comment, Open in v0	Jul 11, 2026 1:32am
eve-docs-4759	Ready	Preview, Comment, Open in v0	Jul 11, 2026 1:32am