fix(operator): report missing gateway CRDs instead of claiming Ready - #173
Open
dcmcand wants to merge 1 commit into
Open
fix(operator): report missing gateway CRDs instead of claiming Ready#173dcmcand wants to merge 1 commit into
dcmcand wants to merge 1 commit into
Conversation
An LLMModel whose AIGatewayRoute, SecurityPolicy or InferencePool could not be applied reported phase Ready with no indication anything was wrong. The reconciler knew: it logged "CRD may not be installed, skipping" and threw the error away, because reconcileRoutingResources and reconcileSecurityPolicies always returned nil. The operator log was the only trace, so the visible symptom was an HTTP 404 or 500 at request time with a healthy-looking model and nothing in ArgoCD to explain it. The PassthroughModel reconciler already solved this, and its own comment named the LLMModel reconciler as the one still logging and moving on. This ports that pattern across: - Both reconcile functions now return per-endpoint outcomes, and reconcileInferencePoolResources returns the pool apply error separately from errors in resources this pack does own. - Each endpoint gets one condition covering its route and its policy together; they fail for the same reason and are fixed by the same action. InferencePool gets its own condition, because it comes from a different CRD bundle: a cluster can have the AI Gateway CRDs but not the Inference Extension's, and that combination crashloops the llm-d End-Point Picker while routing looks fine. - A failed apply downgrades Ready to Degraded. The workload is healthy but unreachable, which is what Degraded describes; Error would overstate it. Pending, Downloading and Starting already say something more specific and are left alone. - The reconcile is requeued so the model heals once the CRDs appear, rather than sitting Degraded until something else triggers it. A missing CRD still does not fail the reconcile. Everything this pack owns is worth converging, and the applies succeed on a later pass. conditionFor, disabledCondition and the condition-type constants move to a new conditions.go now that both controllers use them, and the PassthroughModel phase check uses the shared hasApplyFailure helper rather than re-deriving it from a string literal. Behavior change worth noting for anyone alerting on phase: a model with broken routing that previously reported Ready now reports Degraded. Tests are table-driven for the pure helpers, plus envtest specs that pin the user-visible behavior. envtest installs only this operator's CRDs, so the gateway applies fail with no-kind-match exactly as they do on a cluster where the install runbook's prerequisites were skipped. Both new specs are mutation-verified: they fail when the phase downgrade is disabled and when the pool error is swallowed again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
An
LLMModelwhoseAIGatewayRoute,SecurityPolicyorInferencePoolcould not be applied reported phaseReadywith nothing to indicate a problem. The reconciler knew perfectly well -llmmodel_controller.go:455before this change:reconcileRoutingResourcesthen returnednilunconditionally (//nolint:unparam // error return kept for future extensibility), same forreconcileSecurityPoliciesand theInferencePoolapply. So the operator log was the only trace, and the visible symptom was an HTTP 404 or 500 at request time against a healthy-looking model with nothing in ArgoCD to explain it. This is the failure mode that makes the install runbook's §4 prerequisites so costly to skip.The fix already existed next door
passthroughmodel_controller.go:130solved this, and its comment named the culprit:So this ports that pattern rather than inventing one.
reconcileInferencePoolResourcesreturns the pool apply error separately from failures in resources this pack does own (those stay fatal).InferencePoolReadyis its own condition, because it comes from a different CRD bundle. A cluster can have the AI Gateway CRDs but not the Inference Extension's, and that combination crashloops the llm-d EPP while routing looks fine. Diagnosing that needs the two told apart.Readydegrades toDegradedon a failed apply. The workload is healthy but unreachable, which is whatDegradeddescribes;Errorwould overstate it.Pending/Downloading/Startingalready say something more specific and are untouched.Degradeduntil something else triggers a reconcile.A missing CRD still does not fail the reconcile - everything the pack owns is worth converging.
conditionFor,disabledConditionand the condition constants move to a newconditions.gonow that both controllers use them, and PassthroughModel's phase check uses the sharedhasApplyFailureinstead of re-deriving it from a"ApplyFailed"string literal.A model with broken routing that previously reported
Readynow reportsDegraded. That's the point, but it will move dashboards and can fire alert rules keyed on phase. New conditions are purely additive.Verification
go test ./...green, including 15 envtest specs;golangci-lint run→ 0 issues;go vetandgofmtcleanconditionFor,hasApplyFailure,phaseWithRoutingFailure), including that a disabled endpoint isFalsebut must not count as a failureno matches for kind "InferencePool" in version "inference.networking.k8s.io/v1". The specs first reconcile, then set the Deployment'sreadyReplicasby hand, because nothing runs the Deployment controller in envtest anddeterminePhasewould otherwise returnStartingand short-circuit before any gateway resource is applied.= "Ready", want "Degraded") and the envtest spec failspoolErr→ the envtest spec fails on the InferencePool conditionFollow-up
A troubleshooting entry pointing at these conditions belongs in
installation.md, but that file is being rewritten in #171 and #172. Left out here to avoid the conflict; worth adding once those land.