issues while running hello world code with openmpi (ethernet and OPA)

[puneet336]

Hi ,
I am experiencing issues with MPI applications, Here are the context and detail on the issue : A month ago, we created a test environment with 4 VMs, where all MPI-based applications worked successfully over Ethernet using OpenMPI versions 4.1.2 and 4.1.4 on RHEL 7.9. Recently, we added two physical servers to this cluster, also running RHEL 7.9, with both Omni-Path (OPA) and Ethernet connectivity configured.

However, the problem is that even a simple "hello world" MPI application fails to run on the newly added physical servers.

here is the job script -

#!/bin/bash -x
#BSUB  -o hello-dbg.%J
#BSUB -J hello
#BSUB -n 2
#BSUB -R "rusage[mem=60000] span[ptile=1] affinity[core(1):distribute=pack]"
#BSUB -q testing
#BSUB -m "server3 server4"

module load OpenMPI/4.1.4-GCC-11.2.0
module list
cd /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0
mpicc hello.c
which mpirun
echo $LSB_HOSTS
mpirun  --mca btl_base_verbose 30 /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0/a.out

server1, server2 ( VMS) -
successfulrun.txt

server3 , server4 ( Physical servers) -
failedrun.txt

in both the log files, its not clear that which interface was used specifically for the communication but it seems that the failure happened when ofi btl was initialized. not sure which resources were exhausted.

suspecting there may be issues with the Omnipath setup , i selected all the tcp interfaces on server3 and server4 , but they all failed with same error ,

example -

eth_failrun1.txt
eth_okrun1.txt

despite selecting eth0.2634(network range same as ens192), contrary to my expectation ,

export OMPI_MCA_mtl="^ofi"
mpirun  --mca btl_base_verbose 100 --mca btl_tcp_if_include eth0.2634 /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0/a.out

the ofi interface initialization was attempted -

[server2:3882373] select: initializing btl component ofi
libi40iw-i40iw_vmapped_qp: failed to pin memory for SQ
libi40iw-i40iw_ucreate_qp: failed to map QP

with
export OMPI_MCA_mtl="^ofi,psm2"
export OMPI_MCA_btl="^ofi"
i was able to disable the initialization of ofi, but then it was the usnic

[server3:3203372] select: initializing btl component usnic
libi40iw-i40iw_vmapped_qp: failed to pin memory for SQ
libi40iw-i40iw_ucreate_qp: failed to map QP
[server3:3873721] btl:usnic: disqualifiying myself due

ofi_disable_fail.txt

There may be issues with the OPA , but i am sure that ethernet interfaces are configured correctly.
in order to validate the connectivity between server3 and server4 over interface ens0. i used some tests -

server1 & server2:

-bash-4.2$ ip a s|grep inet|grep ens192
    inet 100.165.150.75/24 brd 100.165.150.255 scope global ens192
-bash-4.2$ ./iperf3 -s -B 100.165.150.75
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 100.165.150.76, port 44857
[  5] local 100.165.150.75 port 5201 connected to 100.165.150.76 port 57216
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.88 GBytes  24.7 Gbits/sec
[  5]   1.00-2.00   sec  2.75 GBytes  23.6 Gbits/sec
[  5]   2.00-3.00   sec  2.62 GBytes  22.5 Gbits/sec
[  5]   3.00-4.00   sec  2.69 GBytes  23.1 Gbits/sec
[  5]   4.00-5.00   sec  2.65 GBytes  22.8 Gbits/sec
[  5]   5.00-6.00   sec  2.65 GBytes  22.8 Gbits/sec
[  5]   6.00-7.00   sec  2.69 GBytes  23.1 Gbits/sec
[  5]   7.00-8.00   sec  2.67 GBytes  23.0 Gbits/sec
[  5]   8.00-9.00   sec  2.60 GBytes  22.3 Gbits/sec
[  5]   9.00-10.00  sec  2.59 GBytes  22.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  26.8 GBytes  23.0 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated by signal Interrupt(2)
-bash-4.2$



-bash-4.2$ ./iperf3 -c 100.165.150.75 -B 100.165.150.76
Connecting to host 100.165.150.75, port 5201
[  5] local 100.165.150.76 port 57216 connected to 100.165.150.75 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.88 GBytes  24.7 Gbits/sec    0   2.05 MBytes
[  5]   1.00-2.00   sec  2.74 GBytes  23.6 Gbits/sec    0   2.05 MBytes
[  5]   2.00-3.00   sec  2.62 GBytes  22.5 Gbits/sec    0   2.05 MBytes
[  5]   3.00-4.00   sec  2.68 GBytes  23.1 Gbits/sec    0   2.05 MBytes
[  5]   4.00-5.00   sec  2.65 GBytes  22.8 Gbits/sec    0   2.05 MBytes
[  5]   5.00-6.00   sec  2.65 GBytes  22.8 Gbits/sec    0   2.05 MBytes
[  5]   6.00-7.00   sec  2.69 GBytes  23.1 Gbits/sec    0   2.05 MBytes
[  5]   7.00-8.00   sec  2.67 GBytes  23.0 Gbits/sec    0   2.05 MBytes
[  5]   8.00-9.00   sec  2.60 GBytes  22.3 Gbits/sec    0   2.05 MBytes
[  5]   9.00-10.00  sec  2.59 GBytes  22.2 Gbits/sec    0   2.05 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  26.8 GBytes  23.0 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  26.8 GBytes  23.0 Gbits/sec                  receiver

iperf Done.

between server3 and server4:

ip a s|grep inet
    inet 100.165.150.100/24 brd 100.165.150.255 scope global eth0.2634
-bash-4.2$ ./iperf3 -s -B  100.165.150.105
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 100.165.150.100, port 38484
[  5] local 100.165.150.105 port 5201 connected to 100.165.150.100 port 35928
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   913 MBytes  7.65 Gbits/sec
[  5]   1.00-2.00   sec   905 MBytes  7.59 Gbits/sec
[  5]   2.00-3.00   sec   994 MBytes  8.33 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.91 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.91 Gbits/sec
[  5]  10.00-10.00  sec  1.12 MBytes  9.44 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  10.8 GBytes  9.29 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------



-bash-4.2$ ./iperf3 -c 100.165.150.105 -B 100.165.150.100
Connecting to host 100.165.150.105, port 5201
[  5] local 100.165.150.100 port 35928 connected to 100.165.150.105 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   914 MBytes  7.66 Gbits/sec  335    140 KBytes
[  5]   1.00-2.00   sec   906 MBytes  7.59 Gbits/sec  290    140 KBytes
[  5]   2.00-3.00   sec   994 MBytes  8.34 Gbits/sec  231    621 KBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.91 Gbits/sec    0    621 KBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.91 Gbits/sec    0    621 KBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.91 Gbits/sec    0    621 KBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.91 Gbits/sec    0    621 KBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0    621 KBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.91 Gbits/sec    0    621 KBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0    621 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec  856            sender
[  5]   0.00-10.00  sec  10.8 GBytes  9.29 Gbits/sec                  receiver

iperf Done.

so you can see that I have already tried disabling specific interfaces and validated iperf for non-MPI communication, which works fine over Ethernet. Is there a checklist or list of basic network-related checks I can perform on the server MPI applications can at least run as expected over Ethernet interfaces?

Thank you in advance for your help!

[bosilca]

Scanning through the output indicates your observation is correct, the OFI BTL (which has a high exclusivity meaning it would rather run alone) prevented the TCP BTL from initializing. However, in most of your tests instead of disabling the OFI BTL (--mca btl ^ofi) you disable the OFI MTL (--mca mtl ^ofi), which will have no impact if OB1 is used (which is the default except for IB). In the case where you did disable the OFI BTL, the UCX PML was somehow selected (there is not enough info to figure out why) but that also failed while creating the QPs.

Try to run with --mca pml ob1 --mca btl tcp,self --mca btl_tcp_if_include 100.165.150.0/24 to force all traffic over TCP.

[puneet336]

Thank you for the response @bosilca

with suggested mca and btl parameters, the application works as expected over ethernet interface -

export OMPI_MCA_pml="ob1"
export OMPI_MCA_btl="tcp,self"
export OMPI_MCA_btl_tcp_if_include="eth0.2634"
export OMPI_MCA_btl_base_verbose="100"
mpirun  /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0/a.out

output -

++ /usr/bin/modulecmd bash list
Currently Loaded Modulefiles:
  1) GCCcore/11.2.0
  2) zlib/1.2.11-GCCcore-11.2.0
  3) binutils/2.37-GCCcore-11.2.0
  4) GCC/11.2.0
  5) numactl/2.0.14-GCCcore-11.2.0
  6) XZ/5.2.5-GCCcore-11.2.0
  7) libxml2/2.9.13-GCCcore-11.2.0
  8) libpciaccess/0.16-GCCcore-11.2.0-BuildNAOS
  9) hwloc/2.7.1-GCCcore-11.2.0
 10) OpenSSL/1.1
 11) libevent/2.1.12-GCCcore-11.2.0
 12) UCX/1.12.1-GCCcore-11.2.0
 13) libfabric/1.15.1-GCCcore-11.2.0
 14) PMIx/4.1.2-GCCcore-11.2.0
 15) UCC/1.0.0-GCCcore-11.2.0
 16) OpenMPI/4.1.4-GCC-11.2.0
+ eval
+ cd /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0
+ mpicc hello.c
+ which mpirun
/install/software/OpenMPI/4.1.4-GCC-11.2.0/bin/mpirun
+ echo server3.example.com server4.example.com
server3.example.com server4.example.com
+ export OMPI_MCA_pml=ob1
+ OMPI_MCA_pml=ob1
+ export OMPI_MCA_btl=tcp,self
+ OMPI_MCA_btl=tcp,self
+ export OMPI_MCA_btl_tcp_if_include=eth0.2634
+ OMPI_MCA_btl_tcp_if_include=eth0.2634
+ export OMPI_MCA_btl_base_verbose=100
+ OMPI_MCA_btl_base_verbose=100
+ mpirun /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0/a.out
[server3:4083172] mca: base: components_register: registering framework btl components
[server3:4083172] mca: base: components_register: found loaded component self
[server3:4083172] mca: base: components_register: component self register function successful
[server3:4083172] mca: base: components_register: found loaded component tcp
[server3:4083172] mca: base: components_register: component tcp register function successful
[server3:4083172] mca: base: components_open: opening btl components
[server3:4083172] mca: base: components_open: found loaded component self
[server3:4083172] mca: base: components_open: component self open function successful
[server3:4083172] mca: base: components_open: found loaded component tcp
[server3:4083172] mca: base: components_open: component tcp open function successful
[server3:4083172] select: initializing btl component self
[server3:4083172] select: init of component self returned success
[server3:4083172] select: initializing btl component tcp
[server3:4083172] btl: tcp: Using interface: eth0.2634
[server3:4083172] btl:tcp: Attempting to bind to AF_INET port 1024
[server3:4083172] btl:tcp: Successfully bound to AF_INET port 1024
[server3:4083172] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[server3:4083172] btl:tcp: examining interface eth0.2634
[server3:4083172] btl:tcp: using ipv6 interface eth0.2634
[server3:4083172] select: init of component tcp returned success
[server4:3422420] mca: base: components_register: registering framework btl components
[server4:3422420] mca: base: components_register: found loaded component self
[server4:3422420] mca: base: components_register: component self register function successful
[server4:3422420] mca: base: components_register: found loaded component tcp
[server4:3422420] mca: base: components_register: component tcp register function successful
[server4:3422420] mca: base: components_open: opening btl components
[server4:3422420] mca: base: components_open: found loaded component self
[server4:3422420] mca: base: components_open: component self open function successful
[server4:3422420] mca: base: components_open: found loaded component tcp
[server4:3422420] mca: base: components_open: component tcp open function successful
[server4:3422420] select: initializing btl component self
[server4:3422420] select: init of component self returned success
[server4:3422420] select: initializing btl component tcp
[server4:3422420] btl: tcp: Using interface: eth0.2634
[server4:3422420] btl:tcp: Attempting to bind to AF_INET port 1024
[server4:3422420] btl:tcp: Successfully bound to AF_INET port 1024
[server4:3422420] btl:tcp: my listening v4 socket is 0.0.0.0:1024
[server4:3422420] btl:tcp: examining interface eth0.2634
[server4:3422420] btl:tcp: using ipv6 interface eth0.2634
[server4:3422420] select: init of component tcp returned success
[server3:4083172] mca: bml: Using self btl for send to [[3559,1],0] on node server3
[server4:3422420] mca: bml: Using self btl for send to [[3559,1],1] on node server4
Hello world from processor server3, rank 0 out of 2 processors
Hello world from processor server4, rank 1 out of 2 processors
[server3:4083172] mca: base: close: component self closed
[server3:4083172] mca: base: close: unloading component self
[server3:4083172] mca: base: close: component tcp closed
[server3:4083172] mca: base: close: unloading component tcp
[server4:3422420] mca: base: close: component self closed
[server4:3422420] mca: base: close: unloading component self
[server4:3422420] mca: base: close: component tcp closed
[server4:3422420] mca: base: close: unloading component tcp

Now, we are shifting focus to resolving Intel Omni-Path (OPA) interface issues. Based on the opainfo output for both servers, the OPA interfaces appear to be functional with "PortState: Active" and "Link Quality: Excellent." Additionally, bandwidth tests using iperf3 over the OPA interfaces (ib0) indicate high data transfer rates, confirming that basic connectivity is working fine

-bash-4.2$ opainfo
hfi1_0:1                           PortGID:0xfe80000000000000:test
   PortState:     Active
   LinkSpeed      Act: 25Gb         En: 25Gb
   LinkWidth      Act: 4            En: 4
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 3,4
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True
   LID: 0x0000004e-0x0000004f       SM LID: 0x0000001a SL: 0
         QSFP Copper,       1m  Hitachi Metals    P/N test-12       Rev 01
   Xmit Data:              28226 MB Pkts:              9447297
   Recv Data:               8529 MB Pkts:              7709560
   Link Quality: 5 (Excellent)



-bash-4.2$ opainfo
hfi1_0:1                           PortGID:0xfe80000000000000:test
   PortState:     Active
   LinkSpeed      Act: 25Gb         En: 25Gb
   LinkWidth      Act: 4            En: 4
   LinkWidthDnGrd ActTx: 4  Rx: 4   En: 3,4
   LCRC           Act: 14-bit       En: 14-bit,16-bit,48-bit       Mgmt: True
   LID: 0x0000003c-0x0000003d       SM LID: 0x0000001a SL: 0
         QSFP Copper,       2m  Hitachi Metals    P/N test-15       Rev 01
   Xmit Data:               3344 MB Pkts:              5153745
   Recv Data:              34386 MB Pkts:             12799930
   Link Quality: 5 (Excellent)

bandwidth test with intel opa -

    inet 192.168.30.56/24 brd 192.168.30.255 scope global ib0
-bash-4.2$ ./iperf3 -c 192.168.30.51 -B 192.168.30.56
Connecting to host 192.168.30.51, port 5201
[  5] local 192.168.30.56 port 60164 connected to 192.168.30.51 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.16 GBytes  18.5 Gbits/sec    0    455 KBytes
[  5]   1.00-2.00   sec  3.04 GBytes  26.1 Gbits/sec    0    708 KBytes
[  5]   2.00-3.00   sec  1.93 GBytes  16.6 Gbits/sec    0    744 KBytes
[  5]   3.00-4.00   sec  1.90 GBytes  16.3 Gbits/sec    0    744 KBytes
[  5]   4.00-5.00   sec  1.91 GBytes  16.4 Gbits/sec    0    744 KBytes
[  5]   5.00-6.00   sec  1.91 GBytes  16.4 Gbits/sec    0    744 KBytes
[  5]   6.00-7.00   sec  1.90 GBytes  16.3 Gbits/sec    0    744 KBytes
[  5]   7.00-8.00   sec  2.62 GBytes  22.5 Gbits/sec    0    819 KBytes
[  5]   8.00-9.00   sec  3.08 GBytes  26.5 Gbits/sec    0    819 KBytes
[  5]   9.00-10.00  sec  3.06 GBytes  26.3 Gbits/sec    0    819 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  23.5 GBytes  20.2 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  23.5 GBytes  20.2 Gbits/sec                  receiver

iperf Done.


-bash-4.2$ ip a s|grep inet|grep ib0
    inet 192.168.30.51/24 brd 192.168.30.255 scope global ib0
-bash-4.2$ ./iperf3 -s -B  192.168.30.51
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.30.56, port 57962
[  5] local 192.168.30.51 port 5201 connected to 192.168.30.56 port 60164
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.15 GBytes  18.5 Gbits/sec
[  5]   1.00-2.00   sec  3.04 GBytes  26.1 Gbits/sec
[  5]   2.00-3.00   sec  1.93 GBytes  16.6 Gbits/sec
[  5]   3.00-4.00   sec  1.90 GBytes  16.3 Gbits/sec
[  5]   4.00-5.00   sec  1.91 GBytes  16.4 Gbits/sec
[  5]   5.00-6.00   sec  1.91 GBytes  16.4 Gbits/sec
[  5]   6.00-7.00   sec  1.90 GBytes  16.3 Gbits/sec
[  5]   7.00-8.00   sec  2.62 GBytes  22.5 Gbits/sec
[  5]   8.00-9.00   sec  3.08 GBytes  26.5 Gbits/sec
[  5]   9.00-10.00  sec  3.07 GBytes  26.3 Gbits/sec
[  5]  10.00-10.00  sec  1.75 MBytes  15.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  23.5 GBytes  20.2 Gbits/sec                  receiver
-----------------------------------------------------------

Results show a bandwidth of approximately 20.2 Gbits/sec over a 10-second interval.

While I understand deeper debugging may require vendor assistance, do you have a checklist or set of steps to validate that the OPA environment is fully prepared for MPI/OpenMPI traffic?

Any guidance would be greatly appreciated as we proceed with troubleshooting this setup.

[bosilca]

iperf3 tests TCP/UDP, exactly the same scenario that the options I provided. I can't help with OPA. @hppritcha ?

[hppritcha]

@cpshereda could someone from Cornelis provide a list of sanity checks to run to verify an OPA network is correctly set up?

[puneet336]

the ulimit locked memory value was 64KB. I raised this value to ulimited and followingissue was resolved with version 4.1.4

[server2:3173708] [17] /home/user1/MyWork/osu/validate/ompi4.1.4-GCC-11.2.0/a.out[0x400869]
[server2:3173708] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
server1.3845577map_hfi_mem: mmap of rcvhdr_bufbase (0xdabbad00040b0000) size 262144 failed: Resource temporarily unavailable
server1.3845577a.out: An unrecoverable error occurred while communicating with the driver
[server1:3845577] *** Process received signal ***
[server1:3845577] Signal: Aborted (6)
[server1:3845577] Signal code:  (-6)
[server1:3845577] [ 0] /lib64/libc.so.6(+0x36400)[0x2b8624b86400]

i tried validating the hello world application using openmpi version 4.1.2
now i am trying to run application using 4.1.2 and i see following in the stdout and facing following issues -

+ mpirun /home/user1/MyWork/osu/validate/changeactivity/pairtest01/try01/a.out
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              server3
  Local adapter:           hfi1_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   glchbs-sp220409-app
  Local device: hfi1_0
--------------------------------------------------------------------------
Hello world from processor server1, rank 0 out of 2 processors
Hello world from processor server2, rank 1 out of 2 processors

though the run went fine , i see that there were some issues while trying to initialize ib0 device (hfi1_0).
How do i confirm which device is being used for running the simulation?
OMPI_MCA_btl_base_verbose=100 shows which devices are being initialized, but it is not very helpful in identifying which devices are being actually used.

please advice.

[puneet336]

Update : the permissions issue was resolved, for the following issue:

--------------------------------------------------------------------------
A call to mkdir was unable to create the desired directory:

  Directory: /scratch/tmp
  Error:     Permission denied

Please check to ensure you have adequate permissions to perform
the desired operation.
--------------------------------------------------------------------------
[server1:04852] [[35992,0],0] ORTE_ERROR_LOG: Error in file util/session_dir.c at line 107
[server1:04852] [[35992,0],0] ORTE_ERROR_LOG: Error in file util/session_dir.c at line 346
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_session_dir failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS

i see that a lot of temp dirs are present in /scratch/tmp owned by different users -

drwx------.   2 user1 group1    4096 Jul  7 16:32 RtmpU2plTf
drwx------.   2 user2 group1        4096 Jul  7 16:33 RtmpkUS8r6
drwx------.   2 user3  group1        4096 Jul  7 16:33 Rtmpo5001x
drwx------.   2 user4  group1        4096 Jul  7 16:33 Rtmp1lkCzo

Identified issues with /scratch/tmp: it appears that two servers out of 80 were not backed by the parallel file system on /scratch/tmp.
As OpenMPI did not have necessary permissions to create directories in /scratch, this was apparently causing the runs to terminate.

[puneet336]

I posted a query yesterday regarding Open MPI 4.1.2 and the warnings/errors related to initializing the OPA hfi1_0 device.

By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              server3
  Local adapter:           hfi1_0
  Local port:              1

I tried setting btl_openib_allow_ib to true, but it didn’t resolve the issue. I understand this is just a warning message and can be ignored, but I want to confirm which interface is being used for communication when this message appears. Specifically, I want to confirm which network interface is being used for communication during the simulation to ensure it is leveraging the high-speed interconnect.

is there a way to clearly determine the exact interface used for communication?
I’ve tried OMPI_MCA_btl_base_verbose=100, but it doesn’t clearly identify the active device.

[puneet336]

i tried
OMPI_MCA_btl='^openib'
and the warning message with openmpi 4.1.2 is gone. . I am trying to use the osu benchmarks to see the impact on the application performance ( incase mpi is selecting ethernet). it would be great if i can determine the interface, pml and btl currently being used directly from openmpi logs.

#10693