For a quick fix, you should be able to set the environment variable NUMBA_NUM_THREADS.

hi @GloriaLiu28 can you check if this still exists with the most recent version of scanpy?

@flying-sheep I noticed we use gearys stats in this spatial_autocorr and some of the gearys related metrics use a decorator from scanpy which might set the parallel=True https://github.com/scverse/scanpy/blob/b058b1792b91b1baaf2e5caf395960005098c2ac/src/scanpy/metrics/_gearys_c.py#L233-L234. Here is the decorator https://github.com/scverse/scanpy/blob/b058b1792b91b1baaf2e5caf395960005098c2ac/src/scanpy/_compat.py#L116. The function is compiled parallel if not _is_in_unsafe_thread_pool() but I am not sure if this means spatial_autocorr will run a parallel numba kernel and I think we need to make sure it doesn't.

It will not be parallelized when the current thread’s name starts with ThreadPoolExecutor, which is the case in dask.

joblib seems to use this: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.ThreadPool

so scanpy’s workaround will probably not work there. we should augment it so it does.

I also found the way joblib adapts to manually reduced thread numbers, which seems like a nice starting point for us: https://github.com/joblib/joblib/blob/ed0806a497268005ad7dad30f79e1d563927d7c6/joblib/_parallel_backends.py#L65

@selmanozleyen Ensure that numba only uses one core here going forward.