Skip to content

Start lifespan loops from an empty context - #20

Merged
ancongui merged 1 commit into
mainfrom
fix/loop-request-context
Oct 8, 2026
Merged

ancongui merged 1 commit into
mainfrom
fix/loop-request-context

Conversation

@ancongui

@ancongui ancongui commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Why

POST {project}/operations/compatibility/check runs under a control request lease (request_execution(control=True) in BodyBoundary._serve). It rescans the inventory, and the new verdict calls open_effects or close_effects inside that request. Those open OutboxLoop, NativeDispatcher, KafkaLoop and ProviderLoop, and RecoveryLoop when it isn't open yet. Each open() calls asyncio.create_task(...), which copies the current context into the long-lived task, including _request_lease.

When the response ends, the request closes that lease. From then on, every execute_pure in those loops fails in _Lease.enter with CatalogError(429, "WV-OPERATION-CAPACITY", "Concurrent execution unavailable"), until the loops are reopened. While the request is still running, the request and the loops share one lease and refuse each other.

This is the normal operator path: readiness is restricted, the operator fixes the cause and presses Check, the rescan turns ready, and the effect loops start inside the request.

The lease isn't the only thing the tasks inherit. They also keep the request's debug lease and the pyfly RequestContext, with the operator's security context and principal, plus correlation, request and tenant ids, transaction state and read-only routing. Nothing in the loops reads the pyfly values today, but a loop opened by the Check button would carry the operator's identity for the rest of the process.

What changes

The owner approved this approach before implementation.

  • Every lifespan-owned long-lived task is created with context=contextvars.Context(), the effectively empty context the lifespan gives them at startup:
    • runtime/scheduler.py: weave-recovery
    • operations/outbox_loop.py: weave-outbox-traversal
    • connectors/dispatcher.py: the native worker supervisors
    • triggers/kafka_loop.py: weave-broker-traversal
    • providers/loop.py: weave-provider-inbox
    • operations/compatibility.py: weave-inventory-freshness. Only the lifespan opens it today; it follows the same rule so that every lifespan task does.
  • The tasks the loops start (outbox delivery turns, Kafka consumer turns and their monitor, native executions) inherit the clean context, so they need no change.
  • operations/execution.py is unchanged.

Resetting _request_lease / _debug_lease inside each loop was considered and rejected. It is a deny list that keeps every other request-scoped value, the pyfly RequestContext included, and every new request-scoped ContextVar would have to be added to it.

Out of scope: tasks the request awaits before it ends (operations/ephemeral.py, operations/event_delivery.py, connectors/kafka.py, and the cleanup tasks in connectors/postgresql.py and connectors/egress.py). They end with the request and must keep its lease.

Tests

tests/unit/operations/test_loop_context.py is parametrized over the six owners. Each case:

  1. Fakes only the database checks in open().
  2. Replaces the owner's long-lived coroutine with a probe.
  3. Opens the owner inside request_execution(control=True) + debug_execution() with a pyfly RequestContext, then ends the request.
  4. Asserts the probe runs execute_pure and sees no RequestContext.

On origin/main all six fail with WV-OPERATION-CAPACITY: Concurrent execution unavailable and an inherited RequestContext. With the change, all six pass.

Verification (local)

  • make check: all requested checks passed (source, docs, docs-site, lint, format, types, unit-contracts, both worker projects, prepare, installed-artifacts).
  • Trial merges with Keep compatibility rescans from refusing work they confirm #19 (f6aca06) and with fix/background-loop-admission (392dd99) are clean. On the combined tree, the new test, test_compatibility_rescan.py, test_recovery_admission.py, test_execution.py and test_transport.py pass (34 tests).

Not run: the integration and e2e suites. The change has no database or network behavior of its own.

Follow-up

This is defect 2 of the background admission work, split out so it can land first. fix/background-loop-admission reserves pure-execution capacity for the loops so they stop competing with requests. fix/capacity-rejection-retry keeps schedules and receipts retryable after a capacity refusal. Neither depends on this change, and none of the three touches the same lines.

A request can open the effect loops. POST .../operations/compatibility/check
rescans the inventory, and the new verdict runs open_effects or close_effects
inside that request. asyncio.create_task copied the request's context into
each long-lived task, including its execution lease. Once the response
ended, that lease was closed, and every execute_pure in those loops failed
with WV-OPERATION-CAPACITY "Concurrent execution unavailable" until the
loops were reopened. The tasks also kept the request's debug lease and its
pyfly request context, which holds the operator's identity.

Create the recovery, outbox, Kafka, provider, native worker, and inventory
freshness tasks with an empty context, the same context the lifespan gives
them at startup. The tasks they start inherit it.
@ancongui
ancongui merged commit cd14323 into main Oct 8, 2026
8 checks passed
@ancongui
ancongui deleted the fix/loop-request-context branch October 8, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant