Discard suspended workflow generators without running cleanup as a workflow exit #178

Activity

1. rmcdaniel

   added

   bugSomething isn't workingSomething isn't working

   kind:defectA public product behavior is incorrectA public product behavior is incorrect

   priority:P2Normal-priority product workNormal-priority product work

   on Oct 20, 2026
2. rmcdaniel commented on Oct 23, 2026

   

   rmcdaniel

   on Oct 23, 2026

   Last edited by rmcdaniel

   MemberAuthor

   More actions

   Current normally installed Python SDK 2.9.0 confirms an observable query-state defect in addition to the original disposal diagnostics.

   Package source 04116871c83e6a502c713789cb452e90917a3438, PyPI wheel SHA-256 1e7547e895485dc8f670bd1b9cc34184b135f249db8da98b16b250bfc6002f73. All 36 installed Python files match that tag and wheel RECORD; no editable install or source overlay. Python 3.12.3, pip 26.2.2, certifi 2026.10.23. Installed source bytes remain unchanged after the observations. These are local SDK API/harness observations, with no Server, database or container.

   Minimal runnable query reproduction after installing durable-workflow==2.9.0:

   from durable_workflow import workflow
   from durable_workflow.workflow import query_state
   
   @workflow.defn(name="disposal-query")
   class Flow:
       def run(self, ctx):
           self.phase = "waiting"
           try:
               yield ctx.schedule_activity("work", [])
           finally:
               self.phase = "cleanup"
   
       @workflow.query("phase")
       def read_phase(self):
           return self.phase
   
   print(query_state(Flow, [], [], "phase"))  # cleanup; expected waiting

   A workflow sets self.phase = "waiting", yields an activity inside try, then assigns self.phase = "cleanup" in finally. Its query simply returns self.phase. With zero completion history, both public query_state(...) and the actual worker preparation helper _prepare_query_state(...) return cleanup, rather than the pending workflow's waiting state. The latter was also invoked after explicit collection. Thus disposal can change the instance before its query handler runs; the problem is broader than duplicate diagnostics. This is not a live Server/query-lease qualification.

   A separate yielded-finally-activity harness case completes its business path but records one actual RuntimeError: generator ignored GeneratorExit through the original Python unraisable hook/stderr. The simple finally-logger control emits the expected one message on this current package; it does not reproduce the older report's duplicate-log count, and does not rescue that historical observation.

   Python 3.10 and 3.12 explicitly define finalization/close as entering pending finalizers with GeneratorExit. A supported fix must account for that language behavior. Keeping a generator only until the query handler returns is insufficient if a returned mutable value still aliases instance state when finalization runs: output ownership/snapshotting also needs a deliberate boundary.

   Product decision: this remains an accepted SDK lifecycle defect, not a docs-only resolution. An implementation needs bounded ownership through query/update consumption, detached observable results before disposal, and separation of discarded SDK logs/commands from genuine success/failure/cancellation cleanup. Cover the existing suspension/context/cancellation matrix, prepared and async query paths, and result aliases before calling it delivered. No implementation, release, Python floor, runtime or cancellation-contract change is included in this investigation. The bounded installation is retired; the small reproducer/provenance/actual outputs are retained for follow-through.
3. rmcdaniel commented on Oct 30, 2026

   

   rmcdaniel

   on Oct 30, 2026

   MemberAuthor

   More actions

   General implementation checkpoint, 2026-10-30. PR #217 still exposes the tests-only remote head 5cf71a2fd3d2618a7cad819c715ebfc754e75dd8; no implementation or release has been pushed.

   The transferred branch now integrates current main normally at local commit c1ba87bb3ce1a04c5942536ee15b03f2287ade53. Local work gives each replay bounded ownership through synchronous/asynchronous query and update result consumption, snapshots escaped mutable values/command payloads, and closes discarded generators quietly. It uses no frame mutation, global unraisable hook or generator cache.

   Actual matched-container evidence:

   • The same 20 inherited controls all fail on the combined source without the implementation, and all 20 pass with it.
   • Expanded lifecycle, update, prepared-local-activity-group and cancellation-context coverage currently gives 103 passes and 3 failures.
   • All three remaining failures are real success/failure/cancellation cleanup logging controls. Actual cleanup reaches its first new durable boundary while logging is still marked as replaying, so the expected real-exit log is absent. The assertion remains unchanged. Cold replay and final outcomes behind that assertion remain unqualified.
   • Mutable command payload escape, abandoned asynchronous query cleanup, discarded-finally logging and direct/delegated yielded-finally disposal controls pass. Initial test-tool images lacked pytest; those attempts ran zero tests and are setup failures.

   Next: correct the real-execution logging boundary while preserving quiet query/validation reconstruction, then qualify the expanded cases, affected worker paths, supported Python versions, full required checks and determinism documentation before publishing implementation to this draft. The acceptance criteria remain open.
4. rmcdaniel commented on Oct 31, 2026

   

   rmcdaniel

   on Oct 31, 2026

   MemberAuthor

   More actions

   General implementation update, October 31: the fix is now backed up in draft #217, current head 22fe336a84ef9356cb1a60b7e71f0156dfcc4c5e, preparing 2.12.2. The earlier three real-cleanup logging failures are fixed with their assertions intact.

   Completed source evidence:

   • Full Python 3.12 nonintegration suite: 3,650 passed / 2 skipped / 125 integration cases deselected, plus Ruff and mypy passing. The expanded lifecycle file passes all 49 cases. Exact 2.12.2 metadata and TLS controls also pass.
   • Corpus policy passes: replay 70 → 72, two append-only fixtures; codec 42 unchanged. The first official worker-query fixture fails on exact main c47ef37 because it returns cleanup rather than waiting, and passes with replay ownership.
   • A second real cleanup defect was found before release: query, validator and accepted-update reconstruction executed an unrecorded ctx.side_effect callback. All three controls fail before the guard; the real-execution/cold-history control passes. Reconstruction now stops at that missing marker, leaving callback execution to the workflow task. The first implementation 687eefd passes the original state fixture but fails the new callback fixture (1 pass / 1 fail); the corrected full suite passes both. No assertion or corpus policy was weakened.
   • Python 3.10 through 3.15 all passed at 687eefd, including package/integration, Sonar and CodeQL. Fresh checks for the final callback guard are running; do not substitute the prior head's results for them.
   • Sonar's two composite test assertions are split without changing their checks. python:S5886 / AaGQOqnOQARXXxZWgL49 is reviewed and resolved as a false positive: dataclasses.replace() preserves the concrete dataclass type by Python's documented contract. Rules, checks and annotations remain enabled.

   The determinism clarification is prepared in Docs #417, exact head ab5024b4a076270b1b04122261d66c0679014629. English and all seven translations, common anchor and reviewed-source hashes are updated; the normal all-eight-locale build and hosted checks pass. It explicitly scopes disposal guarantees to 2.12.2 and stays a draft until that published artifact is qualified. Python can still execute ordinary finalizer code; external cleanup belongs in Activities.

   Next: finish the final head's protected checks/analyzer review, merge normally, publish through the unchanged PyPI producer, qualify the actual installed artifact and lifecycle behavior, then deploy/verify all eight documentation pages. No 2.12.2 tag or package has been published, and #178 remains open.
5. rmcdaniel commented on Oct 31, 2026

   

   rmcdaniel

   on Oct 31, 2026

   MemberAuthor

   More actions

   Delivered in Python SDK 2.12.2, now available on PyPI.

   Queries and update validators retain the pending workflow state. Replay disposal discards its own logs and durable commands quietly, while real success, failure and cooperative cancellation retain their normal cleanup. Read-only reconstruction also stops before executing an unrecorded side-effect callback.

   Source #217 merged as cf56110a8ea809937d9699b666c472be019d2b92, preserving the reviewed tree. All 21 main checks pass, including required Python 3.10–3.15 and connected integration. The normal PyPI producer succeeds. The actual registry wheel's SHA-256 is 56d7c9d763928ed24d4ca19d879bcea329c0cebf6dce2219c8a06e14f128083b; all 40 package files match both the merged source and the fresh installation. With no product source mounted, 133 lifecycle/corpus/TLS tests pass, plus the installed update-conformance shard. The first consumer attempt lacked the declared telemetry SDK test dependency (132 passed / one import error); adding that dependency produces the complete pass without assertion changes. Full source checks pass 3,650 tests, Ruff and mypy; two skipped source cases remain documented, with integration separately green.

   The cleanup guide is live in all eight languages after docs #417, exact merge 304901877ec6f2d03b1328ca0eb83d25fe01bcaa. Main qualification, site publication, and actual Pages deployment all succeed. HTTP checks verify the versioned cleanup anchor and Activity/shield instructions in English, German, Spanish, French, Brazilian Portuguese, Ukrainian, Japanese and Simplified Chinese.

   Ordinary Python finalizer code can physically execute during disposal. External I/O belongs in Activities; the SDK guarantees the documented observable replay behavior. No wire or worker-protocol change.
6. rmcdaniel

   closed this as completedon Oct 31, 2026