test: live multi-provider e2e suite (tag-gated, local only) - #2
Merged
Conversation
Eight end-to-end tests against the real DeepSeek API behind a build tag (never built into CI or a plain go test run): - discovery + ListModels via the production FromEnv path - buffered chat: content, finish reason, live token usage - streaming: deltas fold exactly into the final result, usage via stream_options.include_usage - tool calling buffered + streamed: call emission, argument JSON validity, full follow-up round trip with a RoleTool result - deepseek-reasoner buffered + streamed: ReasoningContent capture, DeltaReasoning deltas, correct answer - error taxonomy: invalid key -> non-retryable 401 *APIError, no credential leakage in error text Credentials: DEEPSEEK_API_KEY from the environment or a repo-root .env (gitignored); the loader never logs file contents and tests skip cleanly when no key resolves. Run: go test -tags e2e -run 'TestE2E' -timeout 15m -v .
Provider-parameterized e2e harness: adding a provider is now one e2eTarget entry (id, base URL, format, key env, default model), with model overrides via <ID>_E2E_MODEL. - generic matrix per provider (subtests): discovery, buffered, streaming (folded deltas == result), tool round trip + streamed assembly, bad-key error taxonomy — each skips cleanly when its key env is absent - openrouter added: custom-provider wiring (WithFormat/WithBaseURL/ WithAPIKey) exercised live for the first time; gpt-4o-mini default - deepseek-specific reasoner arms kept; reasoning-content assertions are now soft probes (DeepSeek elides reasoning_content for some prompts — observed server-side between runs), answer correctness logged not asserted (model IQ is not the SDK contract)
jkyberneees
changed the title
Found live by the Z.ai e2e validation: Z.ai signals 'Insufficient balance or no resource package' as HTTP 429, so the SDK burned the full 8-attempt backoff ladder (~350s) on a permanent, operator-fixable condition. - billingExhausted classifier (insufficient balance / insufficient_quota / no resource package) demotes such 429s to non-retryable at parse time - buffered path returns immediately on billing exhaustion; stream path follows from the Retryable gate - RED-first tests: buffered + stream fail fast in exactly 1 request e2e harness: - Z.ai added as third target (built-in registry, FromEnv path); default model glm-5-turbo (override: ZAI_E2E_MODEL) - discovery logs accessible model IDs (bounded) for diagnostics - live result: deepseek 5/5, openrouter 5/5, zai discovery PASS + chat arms fail fast with the actionable billing error (account needs recharge/resource package)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eight end-to-end tests against the live DeepSeek API, behind a
e2ebuild tag so they never build into CI or a plaingo test ./...run.What it covers
ListModelsvia the productionFromEnvpathstream_options.include_usagedeepseek-reasoner(buffered + streamed):ReasoningContentcapture,DeltaReasoningdeltas*APIError, credentials never leak into error textCredentials
DEEPSEEK_API_KEYfrom the environment or a repo-root.env(now gitignored). Loader never logs file contents; tests skip cleanly when no key resolves. Key was never read or displayed during development.Verified locally
All 8 tests pass against the live API in 6.8s total (observed 2026-09-04). CI is unaffected — the e2e file is excluded by the build tag.