Skip to content

Fix deadlock when threads first resolve lazy Python members concurrently - #154

Merged
quantconnect-kronos[bot] merged 1 commit into
masterfrom
kronos/fix-lazy-member-deadlock
Oct 29, 2026
Merged

quantconnect-kronos[bot] merged 1 commit into
masterfrom
kronos/fix-lazy-member-deadlock

Conversation

@quantconnect-kronos

Copy link
Copy Markdown

Closes #153

What changed and why

Runtime.PyDecimalType and the converter's datetime/tzinfo constructors were Lazy<T> with the default ExecutionAndPublication mode. Their factories run Python code (importing _pydecimal, executing small modules) while the calling thread holds the GIL, and that code can hand the GIL to another thread. If that thread then reaches the same Lazy (e.g. Converter.GetTypeByAlias during overload resolution), it blocks on the Lazy's lock while holding the GIL, and the first thread can never get the GIL back to finish. The two threads deadlock.

The three lazies now use LazyThreadSafetyMode.PublicationOnly, so no lock is held while the factory runs. A second thread runs the factory itself (for the import, Python's own per-module import lock makes it wait with the GIL released), and every caller gets the first published value. The factories already run under the GIL, so running one twice is safe: _pydecimal comes from sys.modules, so the Decimal type is the same object, and a duplicate datetime/GMT class is simply discarded. Behavior when there's no contention is unchanged.

Tests

  • New TestLazyPyMembers.ConcurrentFirstResolutionOfDecimalTypeDoesNotDeadlock. A sys.meta_path finder holds the first thread's _pydecimal import open with the GIL released, so the second thread is guaranteed to reach PyDecimalType (through an overloaded .NET call) while the factory is still running. It passes with the change. Without the change (only Runtime.cs reverted), the test hangs.
  • The converter lazies are created once per process in Converter's static constructor, so the suite can't run them uncreated. They got the same one-line change for the same reason.
  • Ran the full CI steps locally (Python 3.11, .NET 10):
    • dotnet test src/embed_tests/ --filter "FullyQualifiedName!~SetPythonPath": 1028 passed, 8 skipped, 0 failed
    • dotnet test src/embed_tests/ --filter "FullyQualifiedName~SetPythonPath": 1 passed
    • pytest --runtime netcore tests: 478 passed, 23 skipped
    • dotnet test src/python_tests_runner/: 2 passed
  • Tested end to end in a Lean Python backtest with this build. Eight Python threads started together in initialize each made their first overloaded call with a Decimal argument and read self.time/self.utc_time. All returned the expected values, and the backtest completed and traded normally.

🤖 Generated with Claude Code

PyDecimalType and the converter's datetime/tzinfo constructors were
Lazy<T> in ExecutionAndPublication mode. Their factories run Python code
with the GIL held, and that code can hand the GIL to another thread. A
thread that then reached the same Lazy blocked on its lock while holding
the GIL, so the factory could never finish: a deadlock. PublicationOnly
lets that thread run the factory itself; the first published value wins.

Closes #153

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quantconnect-kronos quantconnect-kronos Bot added kronos:develop:working develop: in progress kronos:review Request review labels Oct 29, 2026
@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added kronos:review:queued review: queued kronos:review:working review: in progress and removed kronos:review Request review kronos:review:queued review: queued labels Oct 29, 2026
@quantconnect-kronos-reviewer
quantconnect-kronos-reviewer Bot marked this pull request as ready for review October 29, 2026 13:23

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. This fixes the root cause of #153: the factory no longer holds a lock while it runs Python code that can hand off the GIL. It's the smallest fix that does that, three one-line changes.

What I checked

  • The mechanism. With ExecutionAndPublication, the second thread waits on the Lazy's monitor while it still holds the GIL. With PublicationOnly it runs the factory itself instead. For _pydecimal, it then waits on Python's per-module import lock, which releases the GIL. The finder's find_spec runs inside that lock, so the test exercises exactly this path. Every caller still gets the single published value.
  • Running a factory twice is safe.
    • SetPyMemberTypeOf calls _pyRefs.Add with the GIL held and nothing in between can release it, so the two adds can't race. The losing PyType is still disposed on shutdown with the other refs.
    • For the converter lazies, a second PyModule.FromString("clr_datetime"/"custom_tzinfo", …) reuses the same entry in sys.modules and only rebinds equivalent classes. __reduce_ex__ pickles to the base datetime, so nothing depends on which module object wins.
  • Exceptions are no longer cached. PublicationOnly doesn't cache a factory exception, so a failed first resolution is retried instead of failing every later call. That's no worse than today.
  • The test fails without the fix. I ran TestLazyPyMembers locally (Python 3.11, .NET 10). It passes with the PR. With only Runtime.cs reverted to master, it hung until my timeout killed it. Five more runs of it together with TestConverter all passed (58/58 each time), so it isn't flaky.
  • CI is green: the build and tests on Python 3.8 to 3.14, plus the Lean Python regression and unit test jobs.
  • Evidence and scope. The description reports a full local suite run and a Lean backtest end to end with eight concurrent first callers. The scope matches the issue.

Not blocking, follow-up only: GetModuleLazy (Runtime.cs:375) builds inspect, clrInterop and hexCallable with isThreadSafe: false, and their factories also import Python modules with the GIL held. With LazyThreadSafetyMode.None, a second thread that reaches the value while the factory is still running gets InvalidOperationException, and the Lazy caches that exception. I haven't reproduced this. If it's reachable concurrently, it belongs in its own issue and PR, not this one.

Other repositories: Lean only picks this up after a QuantConnect.pythonnet release and the matching package bump in Lean (currently 2.0.67).

Risk: I set risk:low. The change is three lines and easy to revert. A test reproduces the deadlock and fails without the fix, CI is green including the Lean jobs, and the description reports an end-to-end run. Behavior without contention is unchanged.

@quantconnect-kronos-reviewer quantconnect-kronos-reviewer Bot added risk:low Low risk and removed kronos:review:working review: in progress labels Oct 29, 2026
@quantconnect-kronos
quantconnect-kronos Bot merged commit 25ef934 into master Oct 29, 2026
9 checks passed
@quantconnect-kronos
quantconnect-kronos Bot deleted the kronos/fix-lazy-member-deadlock branch October 29, 2026 13:23
@quantconnect-kronos quantconnect-kronos Bot removed the kronos:develop:working develop: in progress label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:low Low risk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deadlock when two threads first resolve PyDecimalType at the same time

0 participants