kmer prefilter died, crashed in Sequence destructor with memory free checksum error

[tomgoddard]

mmseqs search of a pair of sequences against Colabfold env database fails on Mac ARM with 64 GB of memory with a "prefilter died" message and a Mac crash report showing it crashed in C++ Sequence::~Sequence() destructor in Mac free_list_checksum_botch() and malloc_zone_error() which is associated in other bug reports with "Incorrect checksum for freed object: probably modified after being freed." (apache/arrow#40652).

The crash does not seem to be due to too little memory since the crash also happens using mmseqs option --split-memory-limit 32G on this computer with 64 GB. The crash happens with mmseqs release 18 and release 17.

../mmseqs-r18/bin/mmseqs search msas/prof_res ../colabfold_databases/colabfold_envdb_202108_db msas/res_env msas/tmp3 --num-iterations 3 --db-load-mode 0 -a -e 0.1 --max-seqs 10000 --prefilter-mode 0 --k-score 'seq:96,prof:80' >& mmseqs2.out &

Final stack frames from Mac crash dump:

{"imageOffset":37768,"symbol":"__pthread_kill","symbolLocation":8,"imageIndex":1},
{"imageOffset":26764,"symbol":"pthread_kill","symbolLocation":296,"imageIndex":2},
{"imageOffset":494140,"symbol":"abort","symbolLocation":124,"imageIndex":3},
{"imageOffset":57716,"symbol":"malloc_vreport","symbolLocation":892,"imageIndex":4},
{"imageOffset":235860,"symbol":"malloc_zone_error","symbolLocation":100,"imageIndex":4},
{"imageOffset":114292,"symbol":"free_list_checksum_botch","symbolLocation":40,"imageIndex":4},
{"imageOffset":29176,"symbol":"small_free_list_remove_ptr_no_clear","symbolLocation":964,"imageIndex":4},
{"imageOffset":18076,"symbol":"free_small","symbolLocation":632,"imageIndex":4},
{"imageOffset":1369764,"symbol":"Sequence::~Sequence()","symbolLocation":180,"imageIndex":0},

I've attached the full mmseqs log output and the input mmseqs db files ### (prof_res) and the full Mac crash dump. Colabfold env database is the standard colabfold_envdb_202108.tar.gz (https://wwwuser.gwdg.de/~compbiol/colabfold/colabfold_envdb_202108.tar.gz). The mac is running macOS 15.6.1 model Mac Studio M2 Ultra. I did not see this on Linux, Ubuntu 24.04 with 64 GB of memory, although possibly its malloc implementation does not do the malloc free list checksum detection. My guess would be the memory corruption is on all platforms but some are more tolerant.

prefilter_crash.zip

[Victor-Mihaila]

Hello,

We tried reproducing the issue on a MacOS ARM64 system as well as an Ubuntu24.04 system with an Intel x86_64 CPU and so far we have been unsuccessful as the search terminated without issues on both setups.

Could you please tell us how you built/installed your mmseqs binary?

[tomgoddard]

The mmseqs binary I used is the release 18 precompiled distribution from the mmseqs2 github

https://github.com/soedinglab/MMseqs2/releases/download/18-8cc5c/mmseqs-osx-universal.tar.gz

It may be that the crash is due to a multi-threading issue that depends on the number of cores on the machine. The Mac I used is a Mac Studio M2 Ultra with 24 cores and 64 GB of memory. It reproduces every time, about 5 tries, on this machine. But if it is caused by order of execution in multiple threads it would not be surprising that it would not reproduce on a Mac with different hardware.

I may be able to debug this a bit further. The mmseqs search is running mmseqs prefilter I think 18 or 19 times before it crashes. I could attempt to isolate the test case to that final mmseqs prefilter run that fails. That would reduce the runtime (I recall it takes tens of minutes before it reaches the crash) and then I could try running single threaded to see if it still crashes or if it is sensitive to how many threads are run. In any case with a heap corruption bug in a multi-threaded code it is likely to be a nightmare to try to find the off-by-one array index or non-atomic operation locking problem behind this.

The reason I was running this on a Mac at all is because I develop ChimeraX and am attempting to allow our 20,000 users to do structure prediction with Boltz on their normal work computers using locally computed sequence alignments to avoid the frequent Colabfold server bottlenecks in running large numbers of predictions.

[tomgoddard]

As I noted in my original description I do not observe the crash on Ubuntu 24.04, only on the Mac.

[milot-mirdita]

We are running out of ideas to reproduce the issue on our side. If you have time, could you please try to compile and run with Address Sanitizer-enabled MMseqs2?

There are instructions in the wiki https://github.com/soedinglab/MMseqs2/wiki/MMseqs2-Developer-Guide#sanitizers on how to compile with ASan.

[tomgoddard]

I tried building mmseqs2 on my Mac following the instructions in the user guide (without ASan)

https://mmseqs.com/latest/userguide.pdf

brew install cmake libomp zlib bzip2

./util/build_osx.sh ~/ucsf/MMseqs2 ~/ucsf/mmseqs2_build >& build.txt

and it failed with missing symbol __kmpc_dispatch_deinit

[100%] Linking CXX executable mmseqs
Undefined symbols for architecture x86_64:
  "___kmpc_dispatch_deinit", referenced from:
      Alignment::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long, unsigned long, bool) (.omp_outlined) in libmmseqs-framework.a[2](Alignment.cpp.o)
      doRescorediagonal(Parameters&, DBWriter&, DBReader<unsigned int>&, unsigned long, unsigned long) (.omp_outlined) in libmmseqs-framework.a[11](rescorediagonal.cpp.o)
      fwbw(int, char const**, Command const&) (.omp_outlined) in libmmseqs-framework.a[12](Fwbw.cpp.o)
      AlignmentSymmetry::readInData(DBReader<unsigned int>*, DBReader<unsigned int>*, unsigned int**, unsigned short**, int, unsigned long*) (.omp_outlined) in libmmseqs-framework.a[14](AlignmentSymmetry.cpp.o)
      AlignmentSymmetry::findMissingLinks(unsigned int**, unsigned long*, unsigned long, int) (.omp_outlined) in libmmseqs-framework.a[14](AlignmentSymmetry.cpp.o)
      AlignmentSymmetry::sortElements(unsigned int**, unsigned long*, unsigned long) (.omp_outlined) in libmmseqs-framework.a[14](AlignmentSymmetry.cpp.o)
      ClusteringAlgorithms::execute(int) (.omp_outlined) in libmmseqs-framework.a[16](ClusteringAlgorithms.cpp.o)
      ...
ld: symbol(s) not found for architecture x86_64
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/mmseqs] Error 1
make[1]: *** [src/CMakeFiles/mmseqs.dir/all] Error 2
make: *** [all] Error 2

Full output build.txt is attached. My Mac has an M2 CPU and if I eliminate the x86_64 build from build_osx.sh and just do the ARM build it fails with the same missing symbol __kmpc_dispatch_deinit.

build.txt

The brew installed libomp is version 21.1.3. MacOS version is 15.6.1. XCode version is 15.1. clang version 17.0.0.

$ brew install libomp
Warning: libomp 21.1.3 is already installed and up-to-date.
To reinstall 21.1.3, run:
brew reinstall libomp

I don't mind helping out by building and testing with ASan if the build worked.

[milot-mirdita]

I think the issue should be fixed in latest git. However, we just merged in a large change into MMseqs2, so please use a git commit from a couple days ago:

git clone https://github.com/soedinglab/MMseqs2.git
cd MMseqs2
git checkout befcb1130a73ddba247fe3eaa504e7ffb790d150
mkdir build
cd build
cmake -DHAVE_SANITIZER=1 -DCMAKE_BUILD_TYPE=ASan ..
make -j $(sysctl -n hw.ncpu)

This should work (hopefully).

[tomgoddard]

Forgot to mention I was building release 18 of mmseqs not the current code when I got the symbol not found error.

My plan was to reproduce the crash in my local build, then enable ASan.

Did not want to immediately jump to newer code since the bug is hard to reproduce and may depend on thread order of execution which any code updates may disturb.

[milot-mirdita]

The instructions above should work with 18, just do git checkout 18-8cc5c instead