Crash when loading Python library into process that is redirecting to mimalloc

[mbechard]

Hey, I have a host process that is using mimalloc without issue. As part of it's functionality, it loads python311.dll (Python 3.11) directly into it and uses the CPython API directly to interact with Python. I can see that python311.dll is correctly using mi_malloc() calls for it's allocations as well, so the redirects are working there too.

However, there is one Python module 'open3d' (https://www.open3d.org/) that causes a crash when it's loaded into the process (via Pythons import mechanism). The crash stack is:

        ntdll.dll!RtlReportCriticalFailure�()	Unknown
 	ntdll.dll!RtlpHeapHandleError�()	Unknown
 	ntdll.dll!RtlpHpHeapHandleError�()	Unknown
 	ntdll.dll!RtlpLogHeapFailure�()	Unknown
 	ntdll.dll!RtlFreeHeap�()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbffd8deb8()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd0eeb62()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd0f6f89()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd0f460a()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd0e25df()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd0eb344()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd426265()	Unknown
 	pybind.cp311-win_amd64.pyd!00007ffbfd4260a6()	Unknown
 	python311.dll!00007ffc9ec37c7c()	Unknown
 	python311.dll!00007ffc9ec37e89()	Unknown

It seems to me that pybind.cp311-win_amd64.pyd is linked against the static CRT. Could this cause issues? If so, is there any best solution for loading binaries that use the static CRT while mimalloc is redirecting mallocs in the host process? In theory if the binaries are only allocating/freeing memory that they own I assume that should work, but I guess this isn't the case for Python.

Or alternatively, what is the best way to only use mimalloc for my own binaries and not any 3rd party ones? I guess I'd have to call mi_malloc directly in all my code?

I notice that Python 3.13 is compiled using mimalloc, but there is no open3d release for Python 3.13 yet, so I can't test how things are working in that case.

This crash occurs with both mimalloc 2.1.4 and 3.0.8. If set MIMALLOC_DISABLE_REDIRECT=1, then crash goes away. I've spent a fair amount of time trying the various debug techniques but no luck so far.

You can get a copy of the package from here;
https://files.pythonhosted.org/packages/a3/3c/358f1cc5b034dc6a785408b7aa7643e503229d890bcbc830cda9fce778b1/open3d-0.19.0-cp311-cp311-win_amd64.whl
just rename .whl to .zip to open it. The .pyd in question is in the open3d\cpu directory in the .zip

Thanks!

[malkia]

I wonder if this happens. Without mimalloc, or any global "malloc" redirector, every alloc on WIndows would end up calling HeapAlloc / HeapFree - whether it's /MD (dynamic CRT) or /MT (static CRT).

This way, allocations done in say .dll compiled with static CRT can still be freed (safely) by .dll or .exe linked with dynamic CRT and vice versa

So then with mimalloc or anything else redirecting just malloc/free (new/delete) you would have situations where memory allocated with HeapAlloc, would try to be freed using now mimalloc's free, or allocated with mimalloc would try to be freed with HeapFree.

So TLDR - Before by pure accident, statically compiled CRT and dynamically compiled would still work due them calling compatible HeapAlloc/HeapFree, but with mimalloc that's not the case.

I guess the solution is if mimalloc can override HeapAlloc/HeapFree - but that maybe tricky.

Ideally the Python module should not be compiled against the static CRT library. It really works by magic, and that sounds like a bug!

Which reminds me - we (@ work) may have similar situation - so thanks for bringing it up!

[malkia]

Assuming what I said above is correct, here is what might be happening.

In this file https://github.com/isl-org/Open3D/blob/3550f4eba5a2ae9ebea441bec65b19a1db23d264/CMakeLists.txt#L61

When Open3D itself is build as shared lib (not yet .pyd), but .dll, it'll use Dynamic CRT (/MD), but maybe their pybind sub-project links to it statically, thus forcing (also the static CRT) - these two things are orthogonal - e.g. static library does not imply static CRT and vice versa, but by happenstance it probably did.

I wonder if pybind project, building the .pyd simply force STATIC_WINDOWS_RUNTIME=ON somehow it might work (though my CMake skills are seriously lacking)

[malkia]

Furthermore their github actions workflow force https://github.com/isl-org/Open3D/blob/3550f4eba5a2ae9ebea441bec65b19a1db23d264/.github/workflows/windows.yml#L315

-DSTATIC_WINDOWS_RUNTIME=ON

should probably be

-DSTATIC_WINDOWS_RUNTIME=OFF

[mbechard]

Thank you for the response and raising this with Open3D. That does seem likely to be the issue. I assume Python 3.13 isn't using the auto-redirecting, and is instead explicitly calling mimalloc, so likely it won't suffer from this issue when Open3D provides their package for that. I may try to compile as with your suggestion to confirm if that is the root issue.