Add ClassifierRetryPolicy and keep LLMRetryPolicy as the text-model policy it was

[kaxil]

Follow-up to #73368 and #73450 from a backward-compatibility check of the released 0.9.0 provider against main. Every 0.9.0-style LLMRetryPolicy and LLMBranchOperator pattern from that release's docs was run against main's code with the model mocked and each decision compared with the one 0.9.0 produced.

Summary

#73450 folded a classifier-model mode into LLMRetryPolicy. Two things were wrong with that. It changed what 0.9.0 Dags do on a text model: the model stopped choosing the retry and the delay, custom instructions that named their own categories (the 0.9.0 guide's documented way to customise the policy) had those names silently mapped onto the seven defaults, and ErrorClassification was gone. And it left categories, min_confidence and a confidence gate on a class most users run against a text model, where none of them mean anything.

Retry policies are now one class per layer of a ladder from fully hardcoded to fully model-driven, and the guide is organised that way:

Layer	Class	What decides	Tuned by
Fallback rules	ExceptionRetryPolicy, or fallback_rules on either policy below	RetryRule matches on the exception type, then the task's own retries and retry_delay. No model.	Writing the rules. Always the floor.
Classifier	ClassifierRetryPolicy (new)	The model names one of categories; the table says retry or fail, the delay, and how sure the model has to be. A classifier model such as Jev runs here; a text model can too.	Category descriptions and the confidence bar. No reasoning.
LLM	LLMRetryPolicy (unchanged)	A text model classifies the failure and decides retry and delay from instructions, and explains itself.	The instructions.

• LLMRetryPolicy is the 0.9.0 policy and grows no arguments. Verified decision for decision against the 0.9.0 module for every pattern in the 0.9.0 docs, including the Snowflake example and a prompt that invents a category. ErrorClassification is back, fallback_rules apply verbatim, retry_reason keeps its <category>: <reasoning> shape. Pointed at a classifier model it fails over with a log line naming ClassifierRetryPolicy.
• ClassifierRetryPolicy carries the classifier layer from Let LLMRetryPolicy run on a classifier model with author-defined categories #73450: categories (default DEFAULT_CATEGORIES, the same seven with the same split and delays), min_confidence, the per-category bar, the generated retry_reason line.
• fallback_policy chains the layers. A ClassifierRetryPolicy names a RetryPolicy to consult when its answer is under the bar, reports no confidence, or the classifier call fails; typically an LLMRetryPolicy on a text model. Its RETRY or FAIL is used with the reason prefixed by why the classifier's answer was not (escalated (below_threshold); rate_limit: 429 with Retry-After). Only its RETRY or FAIL ends the chain: a DEFAULT from it, whatever reason it carries, means it decided nothing and the outer fallback_rules apply, then the task's own retry behaviour. So Jev under 0.8 goes to Claude, and if Claude is unreachable the rules decide, including when the inner policy is another classifier. Without min_confidence the classifier's answer is always acted on and fallback_policy is consulted only when the classifier call fails. The example Dag has the chain.

The parse-time warning added in #73450 goes away with the behaviour it warned about.

Live run of the chain

Airflow 3.3, pydantic-ai 2.46.0, typesafe:jev-1.13.0 as the classifier, anthropic:claude-sonnet-5 through the Astro gateway as fallback_policy, a rule RuntimeError -> RETRY 7s as the floor. Two failures, one clear (000606 ... Warehouse 'ANALYTICS_WH' is suspended) and one vague (Upstream job reported status ERROR with no message), through three policies. permanent carries its own bar of 0.95.

Policy	Clear failure	Vague failure
Classifier alone	warehouse_suspended at 1.00, retry 30s	permanent at 0.94, under its bar, so the rule: retry 7s
Classifier with fallback_policy=LLMRetryPolicy(claude)	same, Claude never called	under the bar, Claude called: transient, retry 30s, reason escalated (below_threshold); transient: The error indicates an upstream job failed with a generic ERROR status...
Same chain, LLM connection missing	same	under the bar, Claude call fails with conn_id no_such_connection isn't defined, then the rule: retry 7s, reason classifier answer not applied (below_threshold); rule

One thing the run showed about the bar: on the first attempt the vague message scored permanent at exactly 0.90 against a 0.90 bar and the classifier acted alone; the same message a minute later scored 0.89. The docs already say to set the bar from observed values and that it reduces wrong actions rather than eliminating them; this is what that looks like.

Smaller drifts the check found, fixed here

• On the classifier path, a fallback_rules entry with action=RetryAction.DEFAULT lost its delay and reason; a matched rule is now kept whatever its action.
• An answer missing from categories surfaced as a swallowed KeyError traceback. It now logs what happened and takes the fallback path.
• LLMOperator._push_decision raised AttributeError when the context's task_instance was a plain dict, a shape older tests and hand-built runners use. It now warns and skips the push, as it already did when the key was missing.

Docs

The retry guide opens with the ladder and describes each class under it, with an "Escalating to an LLM" section for the chain. The changelog note #73450 added for the retry policy is gone, since LLMRetryPolicy behaves as it did in 0.9.0 and the new names are additive. Two notes are added for the branching changes: the decision XCom on every run, sorted option order and branches as a template field, and the decision keyword execute_complete grew on four operators. LLMOperator and LLMBranchOperator pass it on resume, so a subclass override of either with the old three-argument signature raises there; the SQL and schema-compare operators accept the keyword but do not pass it yet. The branch guide gets the same "add an option for none of these" guidance the retry guide has.