Manage Python thread states per Julia task

Author: cjdoris

AGENTS.md

@@ -5,7 +5,10 @@
 - **Python tests**:
   - Copy `pysrc/juliacall/juliapkg-dev.json` to `pysrc/juliacall/juliapkg.json` before running (do **not** commit this copy).
   - Execute with `uv run pytest -s --nbval ./pytest` (add `--cov=pysrc` when coverage is needed).
-  - Sometimes `juliapkg` requires Julia 1.10–1.11; `juliaup` already provides 1.11.7 in this environment.
+- Sometimes `juliapkg` requires Julia 1.10–1.11; `juliaup` already provides 1.11.7 in this environment.
+- With Python 3.14, juliapkg's OpenSSL compatibility currently constrains Julia to 1.11 or
+  older. Packages requiring Julia 1.12+ therefore cannot run the Python suite under that
+  Python; use an older Python whose OpenSSL constraint permits Julia 1.12+.
 
 The majority of tests live in the Julia package; Python tests cover functionality that cannot be exercised from Julia (e.g., JuliaCall-specific behavior). Run both suites—typically Julia first—in whichever order makes sense.
 

Project.toml

@@ -34,7 +34,7 @@ PyCall = "1"
 Serialization = "1"
 Tables = "1"
 UnsafePointers = "1"
-julia = "1.10"
+julia = "1.12"
 
 [extras]
 CategoricalArrays = "324d7699-5711-5eae-9e2f-1d82baa6b597"

docs/src/juliacall.md

@@ -157,13 +157,11 @@ caveats.
 
 Most importantly, you can only call Python code while Python's
 [Global Interpreter Lock (GIL)](https://docs.python.org/3/glossary.html#term-global-interpreter-lock)
-is locked by the current thread. You can use JuliaCall from any Python thread, and the GIL
-will be locked whenever any JuliaCall function is used. However, to leverage the benefits
-of multi-threading, you can unlock the GIL while executing any Julia code that does not
-interact with Python.
+JuliaCall borrows the Python thread state which entered Julia and automatically detaches it
+while arbitrary Julia code runs. Nested Python interaction from that Julia code temporarily
+reattaches the same state, and the borrowed state is restored before returning to Python.
 
-The simplest way to do this is using the `_jl_call_nogil` method on Julia functions to
-call the function with the GIL unlocked.
+The historical `_jl_call_nogil` spelling remains available as a compatibility alias:
 
 ```python
 from concurrent.futures import ThreadPoolExecutor, wait
@@ -173,16 +171,9 @@ fs = [pool.submit(jl.Libc.systemsleep._jl_call_nogil, 5) for _ in range(4)]
 wait(fs)
 ```
 
-In the above example, we call `Libc.systemsleep(5)` on four threads. Because we
-called it with `_jl_call_nogil`, the GIL was unlocked, allowing the threads to run in
-parallel, taking about 5 seconds in total.
-
-If we did not use `_jl_call_nogil` (i.e. if we did `pool.submit(jl.Libc.systemsleep, 5)`)
-then the above code will take 20 seconds because the sleeps run one after another.
-
-It is very important that any function called with `_jl_call_nogil` does not interact
-with Python at all unless it re-locks the GIL first, such as by using
-[PythonCall.GIL.@lock](@ref).
+Ordinary calls provide the same automatic resource management, so
+`pool.submit(jl.Libc.systemsleep, 5)` is preferred. PythonCall operations nested inside
+Julia callbacks are safe without explicit region or lock calls.
 
 You can also use [multi-threading from Julia](@ref jl-multi-threading).
 

docs/src/pythoncall-reference.md

@@ -222,8 +222,15 @@ Py(x::MyType) = x.py
 
 ## Multi-threading
 
-These functions are not exported. They support multi-threading of Python and/or Julia.
-See also [`juliacall.AnyValue._jl_call_nogil`](@ref julia-wrappers).
+PythonCall manages Python thread state automatically. These exported macros are optional
+performance and concurrency hints; users normally do not need them for correctness.
+
+```@docs
+@pyregion
+@pyregionbreak
+```
+
+The older `PythonCall.GIL` names remain as compatibility aliases.
 
 ```@docs
 PythonCall.GIL.lock

docs/src/pythoncall.md

@@ -471,42 +471,35 @@ See [Installing Python packages](@ref python-deps).
 
     Multi-threading support is experimental and can change without notice.
 
-From v0.9.22, PythonCall supports multi-threading in Julia and/or Python, with some
-caveats.
-
-Most importantly, you can only call Python code while Python's
-[Global Interpreter Lock (GIL)](https://docs.python.org/3/glossary.html#term-global-interpreter-lock)
-is locked by the current thread. Ordinarily, the GIL is locked by the main thread in Julia,
-so if you want to run Python code on any other thread, you must unlock the GIL from the
-main thread and then re-lock it while running any Python code on other threads.
-
-This is made possible by the macros [`PythonCall.GIL.@unlock`](@ref) and
-[`PythonCall.GIL.@lock`](@ref) or the functions [`PythonCall.GIL.unlock`](@ref) and
-[`PythonCall.GIL.lock`](@ref) with this pattern:
+PythonCall APIs automatically establish the Python thread state they need, so ordinary
+operations can be called from any Julia task or thread without explicit locking. The
+optional [`@pyregion`](@ref) macro amortizes those transitions across straight-line,
+Python-heavy work. Use [`@pyregionbreak`](@ref) around Julia-heavy code which deliberately
+yields, waits, or blocks cooperatively:
 
 ```julia
-PythonCall.GIL.@unlock Threads.@threads for i in 1:4
-  PythonCall.GIL.@lock pyimport("time").sleep(5)
+Threads.@threads for i in 1:4
+  @pyregion pyimport("time").sleep(5)
 end
 ```
 
 In the above example, we call `time.sleep(5)` four times in parallel. If Julia was
 started with at least four threads (`julia -t4`) then the above code will take about
 5 seconds.
 
-Both `@unlock` and `@lock` are important. If the GIL were not unlocked, then a deadlock
-would occur when attempting to lock the already-locked GIL from the threads. If the GIL
-were not re-locked, then Python would crash when interacting with it.
+Both region macros nest arbitrarily, and neither is required for correctness. A nested
+PythonCall operation inside `@pyregionbreak` temporarily re-enters Python automatically.
+On a GIL-enabled Python, attaching a state can block that Julia worker while CPython
+arbitrates access; free-threaded Python uses the same state-management machinery.
 
 With multiple Julia threads you need exactly one interactive thread, see the [FAQ](@ref faq-multi-threading).
 
 You can also use [multi-threading from Python](@ref py-multi-threading).
 
 ### Caveat: Garbage collection
 
-If Julia's GC collects any Python objects from a thread where the GIL is not currently
-locked, then those Python objects will not immediately be deleted. Instead they will be
-queued to be deleted in a later GC pass.
+If Julia's GC collects Python objects while no Python thread state is already attached,
+those objects are queued rather than making a finalizer block while attaching a state.
 
 If you find you have many Python objects not being deleted, you can call
-[`PythonCall.GC.gc()`](@ref) or `GC.gc()` while the GIL is locked to clear the queue.
+[`PythonCall.GC.gc()`](@ref) or `GC.gc()` to clear the queue.

pysrc/juliacall/juliapkg-dev.json

@@ -1,5 +1,5 @@
 {
-    "julia": "^1.10.3",
+    "julia": "^1.12",
     "packages": {
         "PythonCall": {
             "uuid": "6099a3de-0909-46bc-b1f4-468b9a2dfc0d",

pysrc/juliacall/juliapkg.json

@@ -1,5 +1,5 @@
 {
-    "julia": "^1.10.3",
+    "julia": "^1.12",
     "packages": {
         "PythonCall": {
             "uuid": "6099a3de-0909-46bc-b1f4-468b9a2dfc0d",

src/API/exports.jl

@@ -3,6 +3,8 @@ export @py
 export @pyconst
 export @pyeval
 export @pyexec
+export @pyregion
+export @pyregionbreak
 export ispy
 export Py
 export pyabs

src/API/macros.jl

@@ -2,6 +2,8 @@
 macro pyconst end
 macro pyeval end
 macro pyexec end
+macro pyregion end
+macro pyregionbreak end
 
 # Convert
 macro pyconvert end

src/C/context.jl

@@ -21,6 +21,11 @@ A handle to a loaded instance of libpython, its interpreter, function pointers,
 end
 
 const CTX = Context()
+const FINALIZE_HOOK = Ref{Function}(() -> begin
+    if Py_FinalizeEx() == -1
+        @warn "Py_FinalizeEx() error"
+    end
+end)
 
 function _atpyexit()
     if CTX.is_initialized && !CTX.is_preinitialized
@@ -282,9 +287,7 @@ function init_context()
             Py_InitializeEx(0)
             atexit() do
                 CTX.is_initialized = false
-                if Py_FinalizeEx() == -1
-                    @warn "Py_FinalizeEx() error"
-                end
+                FINALIZE_HOOK[]()
             end
         end
         CTX.is_initialized = true