fix(helm): use Recreate strategy for persistence-backed single-replica deployments - #2
Merged
Merged
Conversation
…a deployments Production incident during the #1 rollout: both embedding and reranker default to a single persistence-backed replica sharing one fixed-name ReadWriteOnce PVC. #1 unconditionally switched their update strategy to RollingUpdate, which schedules the new pod before the old one releases the volume — Kubernetes cannot attach an RWO PVC to two pods at once, so the new pod hangs forever on a Multi-Attach error while the old one keeps running, and the rollout never completes. Strategy is now conditional on the same persistence-enabled + replicaCount<=1 guard #1 already uses to disable persistence at replicaCount>1: Recreate whenever a single persistence-backed replica would conflict with itself, RollingUpdate otherwise (replicaCount>1, or persistence disabled). Claude-Session: https://claude.ai/code/session_01A7J6LSaRraBNWn8GtRpJ8t
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fast-follow to #1. During the rollout of #1's chart changes, the reranker deployment got stuck:
FailedAttachVolume/Multi-Attach error for volume ... already used by pod(s) inference-inference-stack-reranker-...— confirmed live in the cluster.Root cause: both
embeddingandrerankerdefault to a single persistence-backed replica sharing one fixed-nameReadWriteOncePVC. #1 unconditionally switched their updatestrategytoRollingUpdate(needed for the newreplicaCount>1horizontal-scaling support), but at the defaultreplicaCount: 1+persistence.enabled: true,RollingUpdateschedules the new pod before the old one releases the volume — an RWO PVC can only be attached to one pod at a time, so the new pod hangs indefinitely while the old one keeps running, and the rollout never completes (this is a hard Kubernetes deadlock, not a transient issue — it does not self-resolve).Fix
strategy.typeis now conditional on the exact samepersistence.enabled && replicaCount<=1guard #1 already uses to disable persistence atreplicaCount>1:Recreatewhenever a single persistence-backed replica would conflict with itself (the default case)RollingUpdateotherwise (replicaCount>1, where persistence is already disabled by fix(router,helm): bounded concurrency, resilient readiness, horizontal scale-out #1's guard, orpersistence.enabled=false)Test plan
helm lint deploy/helm/— cleanhelm templateverified for all three relevant cases: default values (persistence=true, replicaCount=1) →Recreate;replicaCount=3(persistence auto-disabled) →RollingUpdate;persistence.enabled=false,replicaCount=1→RollingUpdatego build ./...sanity-checked anywayhttps://claude.ai/code/session_01A7J6LSaRraBNWn8GtRpJ8t