Skip to content

fix(operator): report missing gateway CRDs instead of claiming Ready - #173

Open
dcmcand wants to merge 1 commit into
mainfrom
fix/llmmodel-surface-missing-crd
Open

fix(operator): report missing gateway CRDs instead of claiming Ready#173
dcmcand wants to merge 1 commit into
mainfrom
fix/llmmodel-surface-missing-crd

Conversation

@dcmcand

@dcmcand dcmcand commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

The bug

An LLMModel whose AIGatewayRoute, SecurityPolicy or InferencePool could not be applied reported phase Ready with nothing to indicate a problem. The reconciler knew perfectly well - llmmodel_controller.go:455 before this change:

if err := r.createOrUpdateUnstructured(ctx, routing.ExternalRoute); err != nil {
    log.Error(err, "failed to reconcile external AIGatewayRoute - CRD may not be installed, skipping")
}

reconcileRoutingResources then returned nil unconditionally (//nolint:unparam // error return kept for future extensibility), same for reconcileSecurityPolicies and the InferencePool apply. So the operator log was the only trace, and the visible symptom was an HTTP 404 or 500 at request time against a healthy-looking model with nothing in ArgoCD to explain it. This is the failure mode that makes the install runbook's §4 prerequisites so costly to skip.

The fix already existed next door

passthroughmodel_controller.go:130 solved this, and its comment named the culprit:

// Gateway kinds tolerate a missing CRD: a failed apply does not fail the whole reconcile. Unlike the LLMModel reconciler (which only logs and moves on), each failure is captured in a status condition and the reconcile is requeued...

So this ports that pattern rather than inventing one.

  • Both reconcile functions return per-endpoint outcomes; reconcileInferencePoolResources returns the pool apply error separately from failures in resources this pack does own (those stay fatal).
  • One condition per endpoint, covering its route and its policy together - they fail for the same reason and are fixed by the same action, so splitting them is noise.
  • InferencePoolReady is its own condition, because it comes from a different CRD bundle. A cluster can have the AI Gateway CRDs but not the Inference Extension's, and that combination crashloops the llm-d EPP while routing looks fine. Diagnosing that needs the two told apart.
  • Ready degrades to Degraded on a failed apply. The workload is healthy but unreachable, which is what Degraded describes; Error would overstate it. Pending/Downloading/Starting already say something more specific and are untouched.
  • Requeued so the model heals once the CRDs appear, instead of sitting Degraded until something else triggers a reconcile.

A missing CRD still does not fail the reconcile - everything the pack owns is worth converging.

conditionFor, disabledCondition and the condition constants move to a new conditions.go now that both controllers use them, and PassthroughModel's phase check uses the shared hasApplyFailure instead of re-deriving it from a "ApplyFailed" string literal.

⚠️ Behavior change

A model with broken routing that previously reported Ready now reports Degraded. That's the point, but it will move dashboards and can fire alert rules keyed on phase. New conditions are purely additive.

Verification

  • go test ./... green, including 15 envtest specs; golangci-lint run0 issues; go vet and gofmt clean
  • Table-driven tests for the pure helpers (conditionFor, hasApplyFailure, phaseWithRoutingFailure), including that a disabled endpoint is False but must not count as a failure
  • envtest specs pin the user-visible behavior. envtest installs only this operator's CRDs, so the gateway applies fail with no-kind-match exactly as on a cluster where §4 was skipped - the test timeline shows the real error: no matches for kind "InferencePool" in version "inference.networking.k8s.io/v1". The specs first reconcile, then set the Deployment's readyReplicas by hand, because nothing runs the Deployment controller in envtest and determinePhase would otherwise return Starting and short-circuit before any gateway resource is applied.
  • Both new specs are mutation-verified, so they cannot pass vacuously:
    • disable the phase downgrade → unit test fails (= "Ready", want "Degraded") and the envtest spec fails
    • restore the swallowed poolErr → the envtest spec fails on the InferencePool condition

Follow-up

A troubleshooting entry pointing at these conditions belongs in installation.md, but that file is being rewritten in #171 and #172. Left out here to avoid the conflict; worth adding once those land.

An LLMModel whose AIGatewayRoute, SecurityPolicy or InferencePool could
not be applied reported phase Ready with no indication anything was
wrong. The reconciler knew: it logged "CRD may not be installed,
skipping" and threw the error away, because reconcileRoutingResources and
reconcileSecurityPolicies always returned nil. The operator log was the
only trace, so the visible symptom was an HTTP 404 or 500 at request
time with a healthy-looking model and nothing in ArgoCD to explain it.

The PassthroughModel reconciler already solved this, and its own comment
named the LLMModel reconciler as the one still logging and moving on.
This ports that pattern across:

- Both reconcile functions now return per-endpoint outcomes, and
  reconcileInferencePoolResources returns the pool apply error separately
  from errors in resources this pack does own.
- Each endpoint gets one condition covering its route and its policy
  together; they fail for the same reason and are fixed by the same
  action. InferencePool gets its own condition, because it comes from a
  different CRD bundle: a cluster can have the AI Gateway CRDs but not
  the Inference Extension's, and that combination crashloops the llm-d
  End-Point Picker while routing looks fine.
- A failed apply downgrades Ready to Degraded. The workload is healthy
  but unreachable, which is what Degraded describes; Error would
  overstate it. Pending, Downloading and Starting already say something
  more specific and are left alone.
- The reconcile is requeued so the model heals once the CRDs appear,
  rather than sitting Degraded until something else triggers it.

A missing CRD still does not fail the reconcile. Everything this pack
owns is worth converging, and the applies succeed on a later pass.

conditionFor, disabledCondition and the condition-type constants move to
a new conditions.go now that both controllers use them, and the
PassthroughModel phase check uses the shared hasApplyFailure helper
rather than re-deriving it from a string literal.

Behavior change worth noting for anyone alerting on phase: a model with
broken routing that previously reported Ready now reports Degraded.

Tests are table-driven for the pure helpers, plus envtest specs that pin
the user-visible behavior. envtest installs only this operator's CRDs,
so the gateway applies fail with no-kind-match exactly as they do on a
cluster where the install runbook's prerequisites were skipped. Both new
specs are mutation-verified: they fail when the phase downgrade is
disabled and when the pool error is swallowed again.
@dcmcand
dcmcand requested a review from marcelovilla August 21, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant