[SAP] Implement graceful shutdown for cinder services#314
Open
hemna wants to merge 2 commits into
Open
Conversation
hemna
force-pushed
the
graceful-shutdown
branch
3 times, most recently
from
5a72074 to
1f69e00
Compare
hemna
force-pushed
the
graceful-shutdown
branch
3 times, most recently
from
4c183ec to
d331087
Compare
hemna
force-pushed
the
graceful-shutdown
branch
2 times, most recently
from
1dd055a to
d2fddd7
Compare
hemna
changed the title
hemna
force-pushed
the
graceful-shutdown
branch
3 times, most recently
from
ca845fc to
a235a58
Compare
Three-phase graceful shutdown that allows in-flight volume operations
(create, delete, clone, snapshot, backup) to complete before the pod
exits during Kubernetes rolling updates.
Phase 1: Send Basic.Cancel to RabbitMQ consumers (no new messages)
without disrupting the _runner greenthread or setting
_consume_loop_stopped (avoids busy-loop CPU starvation).
Phase 2: Block in pool.waitall() until all in-flight RPC handler
greenthreads in the GreenPool complete their operations.
Phase 3: Skip rpcserver.stop()/wait() (hangs on dead AMQP socket).
Process exits cleanly after stop() returns.
Additional mechanisms:
- Worker entry heartbeat in set_workers decorator: touches worker DB
entries every 10s during operations, preventing new pod's init_host
_do_cleanup from resetting in-flight volumes to 'error'.
- do_cleanup freshness check: skips worker entries updated within
service_down_time (60s), only cleans up truly stale/crashed entries.
- Semaphore guard: prevents concurrent stop() calls on same Service.
- Heartbeat continues during drain: service stays 'up' in DB.
- reject_if_draining decorator: rejects new RPC calls during shutdown
so scheduler routes to healthy backends.
Requires:
- dumb-init --single-child (Helm chart change in separate commit)
- terminationGracePeriodSeconds: 900 on pod spec
Tested operations surviving pod termination:
- Volume create from image (41s to 8min drains)
- Volume clone, snapshot create, snapshot delete
- Backup (kill backup pod during stream)
- Backup (kill volume pod during snapshot prep)
- Scheduler rerouting during drain
- Idle shutdown (clean exit <1s)
Change-Id: Icdd28affc73fd34491b656a68410dce8e46264d4
Scsabiii
previously approved these changes
May 15, 2026
Move eventlet imports after stdlib imports (inspect, os, random, etc.) to comply with flake8-import-order (import-order-style = pep8). Change-Id: Icdd28affc73fd34491b656a68410dce8e46264d4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Graceful Shutdown for Cinder Volume Services
Implements three-phase graceful shutdown that allows in-flight volume operations to complete before the pod exits during Kubernetes rolling updates.
How It Works
Phase 1 — Stop new messages (without killing in-flight handlers):
Basic.Canceldirectly for each AMQP consumer tag viaconn.consumer_cancel()conn.stop_consuming()(which causes_runnerbusy-loop starvation)_runnergreenthread stays blocked indrain_events()at 0% CPUPhase 2 — Wait for in-flight operations:
GreenPool.waitall()blocks until all RPC handler greenthreads finishPhase 3 — Clean exit:
rpcserver.stop()/rpcserver.wait()(hangs on dead AMQP socket)stop()returnsAdditional Mechanisms
cinder/objects/cleanable.py):set_workersdecorator spawns a greenthread that touches worker DB entries every 10s during operations. Prevents new pod'sinit_host→_do_cleanupfrom resetting in-flight volumes to 'error'.do_cleanupfreshness check (cinder/manager.py): Skips worker entries updated withinservice_down_time(60s). Only cleans up truly stale/crashed entries.reject_if_drainingdecorator: Rejects new RPC calls during shutdown so scheduler routes to healthy backends.stop()calls on same Service instance.Requirements (separate changes)
dumb-init --single-childon cinder-volume container command — ensures ProcessLauncher parent waits for all children before exitterminationGracePeriodSeconds: 900on pod specbasic.canceltostop_consuming(). Not strictly required by our code path (we callconsumer_cancel()directly) but provides a safety net if other code paths invokestop_consuming()during cleanup.Test Results
See
sap-doc/graceful-shutdown-test-results.mdfor full details.Files Changed
cinder/service.pycinder/manager.pydo_cleanupfreshness check for worker entriescinder/objects/cleanable.pyset_workersdecoratorcinder/volume/manager.pytox.inisap-tools/sap-docfrom flake8doc/source/admin/graceful-shutdown-race-condition.rstsap-doc/graceful-shutdown-test-results.mdNo oslo.messaging source changes required
All changes are self-contained in cinder. We access oslo.messaging internal attributes defensively (
getattrwith fallbacks) but don't modify oslo.messaging source.