fix: make native structured output work with strict providers (#118, #119, #120)

[OsherElhadad]

Summary

Three independent defects, all reported against 0.11.0, made ALTK's native response_format path unusable against strict structured-output providers — including for ALTK's own SPARC metric schemas. Fixed together because they sit on one code path (json_schema_to_pydantic_model → _render_native_schema → the wire) and any one alone still leaves SPARC failing on Bedrock and Azure.

Closes #118, closes #119, closes #120.

Thanks @vz-ibm — the issues were unusually precise, and every root cause and line reference in them held up against the code. Two of the "possible directions" turned out to need adjusting once measured against real providers; that is called out below.

What was wrong, and what changed

#118 — numeric bounds on the wire schema

_CONSTRAINT_ARGS mapped minimum → ge / maximum → le onto the generated Pydantic Field, so all seven SPARC metrics emitted {"type": "integer", "minimum": 1, "maximum": 5} and Bedrock rejected the request: "For 'integer' type, properties maximum, minimum are not supported".

The issue framed this as a genuine tension — the same keywords that keep native output faithful are the ones Bedrock refuses. There is no tension: the generated model is referenced only as the wire value, while _validate runs jsonschema against the original schema dict, so the bounds bought zero local strictness.

But dropping them outright had a cost that only showed up under load: the provider stopped enforcing the range too, and watsonx/openai/gpt-oss-120b promptly returned 0 for a 1-5 score, paying a retry. So a narrowly bounded integer is now emitted as an enum instead — strict providers reject minimum/maximum on an integer but accept enum, verified against Bedrock, Azure, and watsonx. The range is enforced by the provider again, with no rejected keyword and no retry. Wide ranges, half-open bounds, number, and boolean are untouched; minLength/maxLength/minItems/pattern still survive as before.

#119 — the capability gate failed open

Worth recording why the except branch was reached at all: litellm.supports_response_schema does not raise for an unknown model, it returns False. The get_model_info probe that follows is what raised — so "unknown" landed on return True and response_format went to models nobody has capability data for.

The issue proposed routing unknown to the prompt-based path. Measurement said otherwise: watsonx/mistral-large-2512 — the model named in this very issue, and unknown to litellm — honours native output on all seven SPARC schemas at one request per call, as do openai/aws/claude-haiku-4-5 and gateway/proxy strings generally. Since a schema the provider enforces costs one request while one enforced by re-asking costs several, unknown now attempts native.

That is safe because a wrong guess is self-correcting and bounded: a provider that refuses the schema, or a model that answers a native request with empty content (#119's reported symptom — an ignored response_format produces no error, only emptiness), downgrades that call in flight and latches the answer for the client's remaining calls. One request, once — not the retry budget, and not per call. Only a client-side error naming a schema concern counts as a rejection, so a 500 or a rate limit stays retryable and cannot silently disable native output.

I did try going further and ignoring litellm's capability map entirely — on the five watsonx models it reports as unsupported, forced-native scored no worse and once far better (ibm/granite-4-h-small: 1/14 prompt-based vs 13/14 native). Measuring the whole fleet reversed that: under native constrained decoding, the smaller models emit thousands of whitespace-only lines and truncate at the token limit (Expecting ',' delimiter: line 7695), dropping mistral-small-3-1-24b from 7/7 to 1/7 and llama-3-3-70b from 6/7 to 3/7. That is neither a rejection nor an empty response, so no downgrade catches it. The known-negative verdict is therefore still trusted, and the commit that removed it is reverted in this branch — the history keeps both measurements.

It remains imperfect in the other direction: ibm/granite-4-h-small scores 1/21 prompt-based vs 21/21 native, and the default gets that model wrong. So the tri-state native_structured_output knob (None auto / True / False) exists for both directions, settable per client or via the existing chainable configure_validation, and the SPARC README now documents when to reach for it.

#120 — additionalProperties lost in conversion

Two causes, exactly as diagnosed:

1. extra="forbid" was gated on the sub-schema repeating additionalProperties: false, which nested SPARC sub-schemas never do — so nested $defs rendered with the key absent. Forbid is now the default, opted out of only by an explicit additionalProperties: true. Inverting the default is both smaller than threading the parent decision through the recursion and more correct, since threading still gets the wrong answer for a root schema that omits the keyword.
2. Free-form objects rendered as bare dict → additionalProperties: true. BaseValidatingOpenAIClient now defaults free_form_object_as_str = True; with strict: True that is the only shape the API accepts for a free-form object. The alternative of emitting a closed empty object was rejected deliberately — it renders properties: {}, additionalProperties: false, so the model can only ever return {} for correction.corrected_value / tool_call.arguments, and that passes validation, silently discarding the entire correction payload.

A prerequisite bug surfaced here: relax_freeform_object_schema only walked top-level properties, so nested free-form props were never widened. Since json_schema_to_pydantic_model stringifies free-form objects at any depth, the existing free_form_object_as_str path was already broken for SPARC's nested correction fields — a reply with corrected_value: '{"p": 1}' failed validation with '{"p": 1}' is not of type 'object'. Making it recurse is part of this PR; without it, change 2 would have exposed the breakage.

Also fixed: two ways native output was being skipped or broken silently

Validation knobs leaked onto the wire. LiteLLMClientOutputVal.__init__ stashed all lite_kwargs in _lite_kwargs and splatted them into every litellm.completion call, so passing a knob to the constructor put it on the wire: Unrecognized request arguments supplied: free_form_object_as_str, native_structured_output. Pre-existing for the other knobs too; they are now stripped via a shared VALIDATION_KWARGS.

The OpenAI and Azure validating clients never used native output. All four defaulted schema_field=None, so the providers with the strongest native support silently validated by prompt on every call. They now default to "response_format"; pass schema_field=None for the old behaviour. This alone took the direct Azure SDK from never using native to 7/7 native at one request per call.

Verification against real providers

Every number below is from real API calls, not mocks. Credentials: watsonx, Azure OpenAI (direct SDK), and an OpenAI-compatible LiteLLM proxy fronting Bedrock, Azure, and GCP.

Are there any remaining response_format / schema-validation errors?

Census over the seven SPARC schemas, at SPARC's real default retries=3, repeated three times per metric, classifying each failure:

calls	147
REQUEST_REJECTED (provider refused our schema — the #118/#119/#120 class)	0
OUTPUT_INVALID (request accepted; model content failed after all retries)	4

All four were on the prompt-based path and all on one metric (general_value_format_alignment, an out-of-enum issue_type such as MISSING_INFORMATION). Native-strict was 84/84 clean. That is what motivated the native-first work above.

Requests per successful call

Counting actual HTTP requests, ideal = 1 per call:

Target	ok	requests (ideal 7)	native
Bedrock aws/claude-haiku-4-5	7/7	7	yes
Bedrock aws/claude-sonnet-4-5	7/7	7	yes
Azure gpt-4o (proxy)	7/7	7	yes
Azure gpt-4.1 (proxy)	7/7	7	yes
Azure gpt-4o (direct SDK)	7/7	7	yes
watsonx mistral-large-2512	7/7	7	yes
watsonx llama-4-maverick-17b	7/7	7	yes
aws/gpt-oss-120b	7/7	8	yes
watsonx openai/gpt-oss-120b	7/7	7	no
watsonx mistral-medium-2505	7/7	8	no
watsonx mistral-small-3-1-24b	7/7	8	no
gcp/gemini-3-flash-preview	6/7	12	yes
watsonx llama-3-3-70b	7/7	15	no
watsonx ibm/granite-4-h-small	0/7	21	no

Totals: 90/98 succeeded in 128 requests (ideal 98). Seven of the fourteen models run at exactly one request per call, all of them on the native path. granite-4-h-small is the known-negative model that wants native (native_structured_output=True takes it to 21/21); the rest of the overhead is content retries on weaker models.

For comparison, the two policies I measured and rejected: ignoring litellm's capability map scored 84/98 in 135 requests, and the original known-negative-plus-unknown-skip scored 87/98 in 139. The shipped policy is both the most correct and the cheapest of the three.

Wire-level, same schemas, before vs after

Target	Path	Before	After
Bedrock aws/claude-haiku-4-5 (proxy)	native strict, sync + async	0/14	14/14
Azure gpt-4o (proxy)	native strict, sync + async	0/14	14/14
Azure gpt-4o (direct SDK)	strict json_schema	2/7	7/7

Gateway total: 0/35 → 32/35. Forcing the old free-form behaviour back on the direct Azure SDK reproduces the issue's error text verbatim, which pins #120 from both sides:

Invalid schema for response_format 'AutoModel': In context=('properties', 'parameter'),
'additionalProperties' is required to be supplied and to be false.

End-to-end SPARCReflectionComponent

Full static + semantic pipeline over 14 hand-built cases — 5 that must be approved, 9 that must be rejected (wrong city, unit contradicting a stated preference, wrong function for the intent, reversed flight direction, out-of-range value, missing required parameter, wrong type, hallucinated parameter name, non-E.164 phone number, swapped currency codes). Correct decisions, with the expected metric named each time:

Configuration	Mode	Correct	Errors
Bedrock aws/claude-haiku-4-5 native strict	async	14/14	0
Azure gpt-4o (proxy) native strict	async	14/14	0
Azure gpt-4o (proxy) native strict	sync	14/14	0
Bedrock aws/claude-haiku-4-5 auto-detect	async	14/14	0
Azure gpt-4o direct SDK	async	14/14	0
Azure gpt-4o direct SDK	sync	14/14	0
watsonx mistral-large-2512	async	14/14	0
watsonx mistral-large-2512 native override	async	14/14	0
watsonx openai/gpt-oss-120b (reasoning)	async	14/14	0

125/126 correct approve/reject decisions across 9 provider configurations, 0 errors. The single miss is Azure/gpt-4o in sync mode disagreeing on one approve case — a model judgment call on a defensible tool call, not a schema or transport failure; the same configuration scores 14/14 in async.

Non-SPARC schemas

Because SPARC's metrics are one schema family, the same providers were run against five unrelated shapes: a flat Pydantic model, a nested Pydantic model containing an array of sub-models, JSON Schema with enum + bounded integer, four-level-deep nesting with arrays of objects, and nullable + free-form-object fields.

45/45 pass across 9 providers × 5 schemas, all 9 on the native path.

Tests

make test: 402 passed, 8 skipped (baseline measured on main before these changes: 359 passed, 8 skipped — +43 net new tests, no regressions). ruff check, ruff format, and mypy clean; all pre-commit hooks pass including detect-secrets.

Three existing tests asserted behaviour these changes deliberately reverse, and were rewritten with their rationale rather than deleted:

• test_numeric_bounds_survive → test_numeric_bounds_stripped_from_wire_schema
• test_unknown_model_assumes_native_support → test_native_is_attempted_when_not_known_unsupported
• test_watsonx_capability_is_per_model → split into that test plus test_known_unsupported_skips_native, which also asserts the opt-in override

New coverage, including the regression test #120 proposed:

• The seven real SPARC runtime schemas swept for zero additionalProperties: true, zero missing additionalProperties on object schemas, and zero numeric bounds — parametrized per metric, so a failure names the metric.
• Bounded integers render as enum; wide ranges, half-open bounds, number, and boolean do not; out-of-range values still rejected by jsonschema.
• Nested $defs forbid extras without repeating the keyword; explicit additionalProperties: true still honoured.
• relax_freeform_object_schema widens at depth; _validate accepts a nested JSON string and the object literal.
• Schema rejection retries without the kwarg, is remembered for later calls, and unrelated provider errors still propagate (a 500 must not disable native output).
• Empty content under native drops the kwarg.
• All four OpenAI/Azure validating clients default schema_field to "response_format".
• Validation knobs stripped from LiteLLM's replayed kwargs while still applying to the client.

Also ran the -m llm suite against real credentials: 39 passed, 6 failed — 5 ollama (no local server here) and test_azure_openai.py::test_json_schema_validation, which fails identically on unmodified upstream/main with the same credentials (that test passes schema_field=None, so the model free-forms temperature instead of the required temperature_c). Pre-existing and out of scope.

Compatibility

Two behaviour changes worth release notes:

1. The OpenAI/Azure validating clients now send response_format by default. Callers who relied on prompt-based validation there should pass schema_field=None.
2. litellm models that litellm has no metadata for now attempt native output instead of going straight to the prompt path. A model that refuses or ignores it self-corrects after one request per client, but the first such call carries that cost; native_structured_output=False opts out. Models litellm reports as unsupported are unaffected.

Happy to split this into three PRs if you would rather review them separately.