[Bug] fork() called from a process already using GNU OpenMP, this is unsafe.

[quentinblampey]

I got this issue when using fast_array_utils>=1.4.1 and torch DataLoaders with multiple workers. It seems the new fast_array_utils njit decorator is conflicting with torch.

EDIT: see below an MRE.

[quentinblampey]

Minimum reproducible example

I took some time to investigate more, and I came with a minimum reproducible example involving only scanpy/torch/fast_array_utils:

import scanpy as sc
import torch
from anndata import AnnData
from fast_array_utils.conv import to_dense
from torch import Tensor
from torch.utils.data import DataLoader, Dataset


class DummyDataset(Dataset):
    def __init__(self, adata: AnnData) -> None:
        super().__init__()
        self.adata = adata

    def __len__(self) -> int:
        return self.adata.n_obs

    def __getitem__(self, index: int) -> Tensor:
        return torch.tensor(to_dense(self.adata.X[:1], to_cpu_memory=True))  # notice the usage of `to_dense` here


if __name__ == "__main__":
    adata = sc.datasets.pbmc3k()
    sc.pp.normalize_total(adata)  # weirdly, this is needed to reproduce the issue

    dataset = DummyDataset(adata=adata)

    loader = DataLoader(
        dataset,
        batch_size=256,
        shuffle=False,
        drop_last=False,
        num_workers=4,  # >0 workers is important to repoduce the issue
    )

    # this is the line causing the crash (it triggers the creation of batches with workers in parallel)
    for batch in loader:
        break

It crashes with:

Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
RuntimeError: DataLoader worker (pid(s) 1494, 1495, 1496, 1497) exited unexpectedly

OS and versions

My setup is a bit weird: I'm using Coder, hosted by Kubernetes (I can double-check the details with my team if it matters).
It's on Ubuntu 24.04.4 LTS, and I'm using fast_array_utils==1.4.1, anndata==0.12.17, numba==0.65.1, and torch==2.12.1.

Numba threading information:

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : True
+-->Vendor: GNU
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

[flying-sheep]

could you please add a traceback via faulthandler.enable()?

[quentinblampey]

Should I just add this at the beginning of the script? I did that, but it seems not to change the full traceback, see below.

Full traceback

Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Traceback (most recent call last):
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1287, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/queues.py", line 111, in get
    if not self._poll(timeout):
           ~~~~~~~~~~^^^^^^^^^
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/multiprocessing/connection.py", line 1148, in wait
    ready = selector.select(timeout)
  File "/shared/coder/quentin.blampey/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/selectors.py", line 398, in select
    fd_event_list = self._selector.poll(timeout)
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: DataLoader worker (pid 8574) is killed by signal: Terminated. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/shared/coder/quentin.blampey/dev/novae/repro.py", line 37, in <module>
    for batch in loader:
                 ^^^^^^
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 718, in __next__
    data = self._next_data()
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1501, in _next_data
    idx, data = self._get_data()
                ~~~~~~~~~~~~~~^^
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1460, in _get_data
    success, data = self._try_get_data()
                    ~~~~~~~~~~~~~~~~~~^^
  File "/shared/coder/quentin.blampey/dev/novae/.venv/lib/python3.13/site-packages/torch/utils/data/dataloader.py", line 1300, in _try_get_data
    raise RuntimeError(
        f"DataLoader worker (pid(s) {pids_str}) exited unexpectedly"
    ) from e
RuntimeError: DataLoader worker (pid(s) 8574, 8575, 8576, 8577) exited unexpectedly