gh-159485: Fix warnings lock deadlock during interpreter shutdown

[ngoldbaum]

Closes #159485

It looks like the 3.14 backport will need to happen manually but I think the same fix applies there as well if I adjust where the shutdown hook goes.

• Issue: Interpreter shutdown can deadlock when a daemon thread holds the warnings lock #159485

[ZeroIntensity]

An alternative (and simpler) approach would be to hold an interpreter guard while holding the warnings lock.

[ngoldbaum]

I don’t see a way to do that without a larger refactor. warnings.warn() holds the warnings lock while calling user-defined showwarning(), and the issue’s reproducer blocks forever in that callback.

If the guard lasts as long as the lock, shutdown waits forever for the guard. Releasing the guard before the callback doesn’t help if the lock remains held: the callback never returns, so the daemon never releases the lock, and the finalizing thread later blocks on it.

Did you have a different guard placement in mind, or were you suggesting a refactor that releases the lock before calling user code?

[ZeroIntensity]

I was thinking we could acquire a guard before acquiring the lock, and then release it after the lock is acquired. It's fine to call into Python code while a guard is held (and if it isn't, we need to fix that). A thread entering finalization must wait until all daemon threads have released their guards, ensuring that no one holds the lock.

FTR, this kind of issue is one of the reasons PyInterpreterGuard was created in the first place.