mpirun loading wrong libstdc++.so.6

[wadudmiah]

Hi,

I have a simple C++ MPI code (just prints "hello") which I have built with GCC 14.1.0 and the shared libraries it depends on are shown below:

$ ldd hello_cpp.exe
        linux-vdso.so.1 (0x00007ffeec52b000)
        libmpi.so.40 => /apps/ku/amd_9655_cpu/gcc-14_1_0/openmpi/5.0.8/lib/libmpi.so.40 (0x00007f10cc823000)
        libstdc++.so.6 => /apps/ku/amd_9655_cpu/gcc/14.1.0/lib64/libstdc++.so.6 (0x00007f10cc5b1000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f10cc4c9000)
        libgcc_s.so.1 => /apps/ku/amd_9655_cpu/gcc/14.1.0/lib64/libgcc_s.so.1 (0x00007f10cc49c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f10cc293000)
        libopen-pal.so.80 => /apps/ku/amd_9655_cpu/gcc-14_1_0/openmpi/5.0.8/lib/libopen-pal.so.80 (0x00007f10cc167000)
        libucp.so.0 => /apps/ku/amd_9655_cpu/gcc-14_1_0/ucx/1.17.0/lib/libucp.so.0 (0x00007f10cbfed000)
        libuct.so.0 => /apps/ku/amd_9655_cpu/gcc-14_1_0/ucx/1.17.0/lib/libuct.so.0 (0x00007f10cbfa1000)
        libucs.so.0 => /apps/ku/amd_9655_cpu/gcc-14_1_0/ucx/1.17.0/lib/libucs.so.0 (0x00007f10cbdf3000)
        libucm.so.0 => /apps/ku/amd_9655_cpu/gcc-14_1_0/ucx/1.17.0/lib/libucm.so.0 (0x00007f10cbdd4000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x00007f10cbdb8000)
        libpmix.so.2 => /apps/ku/amd_9655_cpu/gcc-14_1_0/pmix/5.0.8/lib/libpmix.so.2 (0x00007f10cbb8b000)
        libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x00007f10cbb82000)
        libevent_core-2.1.so.7 => /apps/ku/amd_9655_cpu/gcc-14_1_0/libevent/2.1.12/lib/libevent_core-2.1.so.7 (0x00007f10cbb4c000)
        libevent_pthreads-2.1.so.7 => /apps/ku/amd_9655_cpu/gcc-14_1_0/libevent/2.1.12/lib/libevent_pthreads-2.1.so.7 (0x00007f10cbb47000)
        libhwloc.so.15 => /usr/lib64/libhwloc.so.15 (0x00007f10cbaee000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f10ccbc3000)

You can see that libstdc++.so.6 is resolved at /apps/ku/amd_9655_cpu/gcc/14.1.0/lib64. However, when I try to run that application, it instead loads /lib64/libstdc++.so.6, which is obviously incorrect:

$ mpirun -n 2 ./hello_cpp.exe
./hello_cpp.exe: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ./hello_cpp.exe)
./hello_cpp.exe: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ./hello_cpp.exe)
--------------------------------------------------------------------------
prterun detected that one or more processes exited with non-zero status,
thus causing the job to be terminated. The first process to do so was:

   Process name: [prterun-cn-0902-17-176777@1,0]
   Exit code:    1
--------------------------------------------------------------------------

I do not know why it is picking up the wrong libstdc++.so.6. When I run it without mpirun, it executes without any issues. Any help will be greatly appreciated.

Regards,

[wadudmiah]

I am using Open MPI 5.0.8

[rhc54]

Just to be clear: mpirun isn't loading the wrong library (at least, not per your provided info). It is your app that is loading it.

Your app also appears to have several dependencies that are coming from non-amd places:

        libm.so.6 => /lib64/libm.so.6 (0x00007f10cc4c9000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f10cc293000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x00007f10cbdb8000)
        libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x00007f10cbb82000)
        libhwloc.so.15 => /usr/lib64/libhwloc.so.15 (0x00007f10cbaee000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f10ccbc3000)

So it looks like you have some LD_LIBRARY_PATH confusion.

[bosilca]

Your LD_LIBRARY_PATH is not properly propagated to the new app. To check this out you have two ways:

1. mpirun -n 1 printenv : to see the remove environment
2. mpirun -n 2 -x LD_LIBRARY_PATH ./hello_cpp.exe to propagate your local LD_LIBRARY_PATH onto the remote

[wadudmiah]

Thanks for the replies. I have found the issue, but I don't understand it. The directory which mpirun was stored in had the following permissions:

drwxrwsr-x 2 9006 2001 12288 Oct 30 13:04 /apps/ku/amd_9655_cpu/gcc-14_1_0/openmpi/5.0.8/bin/

I have removed the s in the group write permission to:

drwxrwxr-x 2 9006 2001 12288 Oct 30 13:04 /apps/ku/amd_9655_cpu/gcc-14_1_0/openmpi/5.0.8/bin/

It is now working. I need to understand what this s sticky bit does. Thanks for your help!