Devito using mpi4py based on OpenMPI off the hpc-x 2.9 distro

[drmichaelt7777]

Hello,

I have been building mpi4py using OpenMPI on RHEL7.9 off the HPC-X 2.9 package on MOFED 5.1. I need to use mpi4py for Devito types of applications.

HPCX v2.9.0 brings in components at these versions:
ompi-535358e gitclone (535358e) (Open MPI) 4.1.2a1
hcoll-e1e5680 4.7.3199 (e1e5680)
nccl_rdma_sharp_plugin-c81cc4f 1.0 (c81cc4f)
sharp-4f31054 2.5.0 (4f31054)
ucx-6031c98 1.11.0 (6031c98)
ofed MLNX_OFED_LINUX-5.1-0.6.6.0

Is in your experience this MPI stack successful with mpi4py ?

Are any items I need to watch out for or do specifically for this particular stack?

I am receiving SIGSEGV at run time but it is not clear if they are generated by the ompi+mpi4py stack.

[labtusc0s000002:109322:0:109322] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2adf03e03438)
[labtusc0s000002:109322:1:109889] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2adf03e037b0)
[labtusc0s000002:109320] *** Process received signal ***
[labtusc0s000002:109320] Signal: Segmentation fault (11)
[labtusc0s000002:109320] Signal code: Invalid permissions (2)
[labtusc0s000002:109320] Failing at address: 0x2b9ca1199780
[labtusc0s000002:109320:0:109907] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2b9ca1199af8)
[1631992035.290020] [labtusc0s000002:109314:0]           debug.c:1377 UCX  WARN  ucs_debug_disable_signal: signal 8 was not set in ucs
[1631992035.290098] [labtusc0s000002:109314:1]        spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
[1631992035.290103] [labtusc0s000002:109314:2]           debug.c:1377 UCX  WARN  ucs_debug_disable_signal: signal 11 was not set in ucs
[1631992035.290352] [labtusc0s000002:109314:2]        spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
[labtusc0s000002:109314:1:109898] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2af198f9ba08)
[labtusc0s000002:109314:2:109896] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2af198f9b7b8)
[1631992035.290357] [labtusc0s000002:109314:3]           debug.c:1377 UCX  WARN  ucs_debug_disable_signal: signal 11 was not set in ucs
[1631992035.290441] [labtusc0s000002:109314:0]           debug.c:1377 UCX  WARN  ucs_debug_disable_signal: signal 11 was not set in ucs
[labtusc0s000002:109314:0:109314] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2af198f9b690)
[labtusc0s000002:109314:3:109897] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2af198f9b8e0)
[1631992035.290398] [labtusc0s000002:109314:1]        spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
[1631992035.290441] [labtusc0s000002:109314:3]        spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
[1631992035.290398] [labtusc0s000002:109314:2]        spinlock.c:29   UCX  WARN  ucs_recursive_spinlock_destroy() failed: busy
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

/tmp/devito-jitcache-uid244579/646cce8ec0a18364be4242ff578d45b82303c270.c: [ pad_vp._omp_fn.0() ]
      ...
       39   /* Begin section0 */
       40   START_TIMER(section0)
       41   for (int abc_x_l = x_m; abc_x_l <= abc_x_l_ltkn + x_m - 1; abc_x_l += 1)
==>    42   {
       43     #pragma omp parallel num_threads(nthreads)
       44     {
       45       #pragma omp for collapse(1) schedule(static,1)


/tmp/devito-jitcache-uid244579/646cce8ec0a18364be4242ff578d45b82303c270.c: [ pad_vp._omp_fn.0() ]
      ...
       39   /* Begin section0 */
       40   START_TIMER(section0)
       41   for (int abc_x_l = x_m; abc_x_l <= abc_x_l_ltkn + x_m - 1; abc_x_l += 1)
==>    42   {
       43     #pragma omp parallel num_threads(nthreads)
       44     {
       45       #pragma omp for collapse(1) schedule(static,1)


/tmp/devito-jitcache-uid244579/646cce8ec0a18364be4242ff578d45b82303c270.c: [ pad_vp._omp_fn.0() ]
      ...
       39   /* Begin section0 */
       40   START_TIMER(section0)
       41   for (int abc_x_l = x_m; abc_x_l <= abc_x_l_ltkn + x_m - 1; abc_x_l += 1)
==>    42   {
       43     #pragma omp parallel num_threads(nthreads)
       44     {
       45       #pragma omp for collapse(1) schedule(static,1)

==== backtrace (tid: 109898) ====
 0 0x00000000000009c0 pad_vp._omp_fn.0()  /tmp/devito-jitcache-uid244579/646cce8ec0a18364be4242ff578d45b82303c270.c:42
 1 0x00000000000a171c __kmp_GOMP_microtask_wrapper()  ???:0
 2 0x000000000013a1a3 __kmp_invoke_microtask()  ???:0
 3 0x00000000000b89c3 __kmp_invoke_task_func()  ???:0
 4 0x00000000000b7a22 __kmp_launch_thread()  ???:0
 5 0x000000000013ae2c _INTERNAL69a793ca::__kmp_launch_worker()  ???:0
 6 0x0000000000007ea5 start_thread()  pthread_create.c:0
 7 0x00000000000fe9fd __clone()  ???:0
=================================
==== backtrace (tid: 109896) ====
 0 0x00000000000009c0 pad_vp._omp_fn.0()  /tmp/devito-jitcache-uid244579/646cce8ec0a18364be4242ff578d45b82303c270.c:42
 1 0x00000000000a171c __kmp_GOMP_microtask_wrapper()  ???:0
 2 0x000000000013a1a3 __kmp_invoke_microtask()  ???:0
 3 0x00000000000b89c3 __kmp_invoke_task_func()  ???:0
 4 0x00000000000b7a22 __kmp_launch_thread()  ???:0
 5 0x000000000013ae2c _INTERNAL69a793ca::__kmp_launch_worker()  ???:0
 6 0x0000000000007ea5 start_thread()  pthread_create.c:0
 7 0x00000000000fe9fd __clone()  ???:0
=================================

On that note, has anyone tried to use IntelMPI with Devito?

Thanks!
Michael