Feat use structured output for agentic mode when available#255
Open
valdis wants to merge 13 commits into
Open
Conversation
Agentic adapters previously bypassed capability detection and always returned free-text JSON, causing parse failures when models included trailing commas or truncated output. Wires structured output (via each SDK's native API) through all three adapters using the same detectCapabilities() path already used by the file-by-file pipeline. - Resolve structuredDialect once in AgenticExecutor via ConfigService + detectCapabilities(), pass to adapters via AgentAdapterParams - AnthropicAdapter: pass outputFormat JSON schema when not unstructured, read structured_output from result message - OpenAIAdapter: pass outputType (z.object wrapper) when not unstructured, return structuredOutput from finalOutput - ConfigurableAgentAdapter: use output.structured mode when not unstructured, read event.structured from FinalEvent - Executor branches on result.structuredOutput (skip text parsing) vs text fallback; throws when non-empty text contains no JSON at all - Restore trailing-comma fix in result-parser fallback path (dropped in QUALOPS-18) - Add isUnstructured(dialect) helper to capabilities.ts to avoid magic literals - Validate --diff-filter param in listChangedFiles against allowlist
…red fallback Agentic mode cannot produce ReviewIssue[] from prose output — the pipeline-executor already intercepts unstructured models at line 59 and routes them to the prose pipeline before any agentic job runs. The isUnstructured() guards inside the adapters were therefore dead code and implied a fallback path that can never be exercised. - Remove structuredDialect from AgentAdapterParams — adapters always use structured output and no longer need to inspect the dialect - Remove isUnstructured() imports and conditional branches from all three adapters (anthropic, openai, configurable-agent) - Add explicit invariant check in AgenticExecutor: if detectCapabilities returns 'unstructured', throw immediately with a clear message instead of silently producing zero findings - Update tests to reflect always-structured adapter behaviour
…names - agentic-executor: remove duplicate soft-error test; fix onToolCall test to assert the callback is passed (not a local array populated by the test itself); strengthen provider test to assert a valid AIProviderName - anthropic-adapter: remove standalone error_max_turns test duplicated by it.each table; strengthen it.each to also assert output === ''; rename 'logs' tests to describe what is actually verified - openai-adapter: fix 'continues without bash tool' to assert structuredOutput rather than output === '' (which is always empty now)
…ard in agentic executor Log first 500 chars of structured_output in AnthropicAdapter on par with the text-fallback log already present. Remove the unreachable isUnstructured() invariant check from AgenticExecutor — pipeline-executor.ts already routes unstructured models to the prose pipeline before any agentic job is dispatched, making this check dead code by design.
… compatibility
Both the Anthropic Claude Agent SDK and Vercel AI SDK (used by configurable-agent)
require a root object schema for structured output — a root array schema causes
the SDK to fall back to plain text output. Wrap ReviewIssuesSchema in
{ issues: [...] } for both adapters and unwrap on the way out, consistent with
the existing OpenAI adapter pattern.
…ed output Anthropic's constrained-decoding structured output (output_config.format / --json-schema) does not support numeric/string constraints (minimum, maximum, multipleOf, minLength, maxLength). When present, the SDK silently falls back to plain-text output instead of structured_output. Add stripUnsupportedConstraints option to schemaToJsonSchema() and enable it in AnthropicAdapter and ConfigurableAgentAdapter. The Zod schema retains its validation constraints for runtime use — only the JSON Schema emitted to the SDK has them stripped.
Picks up Claude Code CLI 2.1.179. No structured output fixes in this release — outputFormat with MCP tools still falls back to text parsing. Upgrade tracked for when SDK adds support.
…ponses
- extractJsonText: handle unclosed code fences (truncated responses where
closing ``` is missing) by extracting content after the opening fence marker
- recoverPartialJsonArray: apply trailing-comma fix and escapeUnescapedControlChars
to each extracted {..} chunk, restoring robustness lost in PR #145 refactor
- review-issue schema: restore .min(1) on description and .int().min(1).max(10)
on confidence so Zod runtime validation is preserved (constraints are stripped
only from the JSON Schema emitted for structured output via stripUnsupportedConstraints)
…tructured output The Claude Code CLI falls back to plain text when the json-schema passed to --json-schema contains a \$schema field (draft-2020-12 URI). Stripping it causes the CLI to enforce constrained decoding and populate structured_output on the result message. Confirmed empirically: with \$schema present structured_output is absent; without it structured_output is populated correctly.
…tput Both Anthropic structured-output dialects (anthropic-output-config and tool_use fallback) rejected root z.array() schemas. Apply the existing wrapArrayRootSchema / unwrapArrayRootResult utilities (already used by OpenAICompatibleProvider) so all callers — FileReviewer, ValidationResolver, DeduplicationResolver — work without any call-site changes. Add unit tests for array-root.ts which had no coverage.
…configurable-agent - openai-compatible-provider: switch from zodResponseFormat (strict) to non-strict json_schema response_format; strict mode rejects .optional() fields unless also .nullable(), which several review schemas rely on. Validate with zod ourselves. Apply wrapArrayRootSchema/unwrapArrayRootResult for array-rooted schemas. - structured/index.ts: export array-root utilities (ARRAY_ROOT_WRAP_KEY, isArrayRootSchema, wrapArrayRootSchema, unwrapArrayRootResult) - configurable-agent-adapter: add supportsStructuredOutputs:true so the AI SDK sends json_schema constrained decoding instead of loose json_object; replace ad-hoc wrapper.issues access with robust unwrapStructuredIssues helper
QualOps Code Quality AnalysisStatus: Summary
🟠 High Issues (1)
📊 Full ReportPowered by QualOps |
If structuredOutput is set but not an array, the adapter's unwrapping logic failed to produce the expected shape. Silently discarding to zero issues would hide a real bug. Throw instead so the failure is visible.
…sonContent check - extract-json: restrict unclosed-fence extraction to ```json only; bare ``` blocks (python, bash, etc.) fall through to substring search - anthropic-adapter: warn and leave structuredOutput undefined when structured_output is present but issues key is missing, so agentic-executor can fall through to text recovery instead of throwing - agentic-executor: use extractJsonText() for hasJsonContent instead of naive string includes — prose with '[' no longer suppresses parse errors
sebastianwessel
approved these changes
Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires up native structured output across all review paths — agentic adapters and the file-by-file pipeline — so the model is constrained to return valid JSON instead of being parsed heuristically from free text.
Root causes fixed
z.array()schemas directly to provider SDKs — both OpenAIzodResponseFormatand AnthropiczodOutputFormatreject a root array with "Root schema must have type: object".$schemafield ("https://json-schema.org/draft/2020-12/schema") in the JSON Schema sent to the Claude Code CLI caused it to silently fall back to plain text instead of enforcing constrained decoding.Changes
Agentic structured output (Anthropic Claude Agent SDK)
ReviewIssuesSchemain a root object{ issues: [...] }— required by the SDK's constrained decodingminimum,maximum,minLength,maxLength, etc.) that cause silent fallback to text$schemafield from the schema passed tooutputFormat— the CLI rejects constrained decoding when it is present (confirmed empirically)Plaintext JSON recovery (fallback path)
extractJsonText: handle unclosed code fences (truncated responses where the closing```is missing)recoverPartialJsonArray: apply trailing-comma fix andescapeUnescapedControlCharsto each extracted{...}chunk, restoring robustness lost in the PR feat: native LLM structured responses via zod-driven complete #145 refactorArray-root wrapping for all providers
wrapArrayRootSchema/unwrapArrayRootResultutilities insrc/ai/shared/structured/array-root.tsAnthropicProvider: apply wrapping in bothanthropic-output-configandtool_usefallback pathsOpenAICompatibleProvider: switch fromzodResponseFormatstrict mode to non-strictjson_schemaresponse format (strict mode rejects.optional()fields unless also.nullable()); apply array-root wrappingConfigurableAgentAdapter: addsupportsStructuredOutputs: trueso the Vercel AI SDK sendsjson_schemaconstrained decoding instead of loosejson_objectTest plan
npm test)