Repository navigation
Start lifespan loops from an empty context - #20
Merged
Merged
Conversation
A request can open the effect loops. POST .../operations/compatibility/check rescans the inventory, and the new verdict runs open_effects or close_effects inside that request. asyncio.create_task copied the request's context into each long-lived task, including its execution lease. Once the response ended, that lease was closed, and every execute_pure in those loops failed with WV-OPERATION-CAPACITY "Concurrent execution unavailable" until the loops were reopened. The tasks also kept the request's debug lease and its pyfly request context, which holds the operator's identity. Create the recovery, outbox, Kafka, provider, native worker, and inventory freshness tasks with an empty context, the same context the lifespan gives them at startup. The tasks they start inherit it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
POST {project}/operations/compatibility/checkruns under a control request lease (request_execution(control=True)inBodyBoundary._serve). It rescans the inventory, and the new verdict callsopen_effectsorclose_effectsinside that request. Those openOutboxLoop,NativeDispatcher,KafkaLoopandProviderLoop, andRecoveryLoopwhen it isn't open yet. Eachopen()callsasyncio.create_task(...), which copies the current context into the long-lived task, including_request_lease.When the response ends, the request closes that lease. From then on, every
execute_purein those loops fails in_Lease.enterwithCatalogError(429, "WV-OPERATION-CAPACITY", "Concurrent execution unavailable"), until the loops are reopened. While the request is still running, the request and the loops share one lease and refuse each other.This is the normal operator path: readiness is restricted, the operator fixes the cause and presses Check, the rescan turns ready, and the effect loops start inside the request.
The lease isn't the only thing the tasks inherit. They also keep the request's debug lease and the pyfly
RequestContext, with the operator's security context and principal, plus correlation, request and tenant ids, transaction state and read-only routing. Nothing in the loops reads the pyfly values today, but a loop opened by the Check button would carry the operator's identity for the rest of the process.What changes
The owner approved this approach before implementation.
context=contextvars.Context(), the effectively empty context the lifespan gives them at startup:runtime/scheduler.py:weave-recoveryoperations/outbox_loop.py:weave-outbox-traversalconnectors/dispatcher.py: the native worker supervisorstriggers/kafka_loop.py:weave-broker-traversalproviders/loop.py:weave-provider-inboxoperations/compatibility.py:weave-inventory-freshness. Only the lifespan opens it today; it follows the same rule so that every lifespan task does.operations/execution.pyis unchanged.Resetting
_request_lease/_debug_leaseinside each loop was considered and rejected. It is a deny list that keeps every other request-scoped value, the pyflyRequestContextincluded, and every new request-scopedContextVarwould have to be added to it.Out of scope: tasks the request awaits before it ends (
operations/ephemeral.py,operations/event_delivery.py,connectors/kafka.py, and the cleanup tasks inconnectors/postgresql.pyandconnectors/egress.py). They end with the request and must keep its lease.Tests
tests/unit/operations/test_loop_context.pyis parametrized over the six owners. Each case:open().request_execution(control=True)+debug_execution()with a pyflyRequestContext, then ends the request.execute_pureand sees noRequestContext.On
origin/mainall six fail withWV-OPERATION-CAPACITY: Concurrent execution unavailableand an inheritedRequestContext. With the change, all six pass.Verification (local)
make check: all requested checks passed (source, docs, docs-site, lint, format, types, unit-contracts, both worker projects, prepare, installed-artifacts).f6aca06) and withfix/background-loop-admission(392dd99) are clean. On the combined tree, the new test,test_compatibility_rescan.py,test_recovery_admission.py,test_execution.pyandtest_transport.pypass (34 tests).Not run: the integration and e2e suites. The change has no database or network behavior of its own.
Follow-up
This is defect 2 of the background admission work, split out so it can land first.
fix/background-loop-admissionreserves pure-execution capacity for the loops so they stop competing with requests.fix/capacity-rejection-retrykeeps schedules and receipts retryable after a capacity refusal. Neither depends on this change, and none of the three touches the same lines.