MLMG failed on HPC

[potatoslc]

Hi everyone, I am trying to run a simulation with assembled diagFramePlanes. I did a low resolution check on local computer and everything works as normal and I decided to use HPC to run a full length of simulation and error comes out after the initialization:
Est. time step - Conv: 2.731568278e-08, divu: 6.84421758e-05 STEP [-1] - Time: 0, dt 2.731568278e-10 SDC iter [1] Iterative solve for deltaT DeltaT solve norm [0] = 0.001000093801 DeltaT solve norm [1] = 0.04287705034 DeltaT solve norm [2] = 1.191272294e-07 DeltaT solve norm [3] = 9.261031731e-13 SDC iter [2] amrex::Abort::63::MLMG failed. !!! SIGABRT amrex::Abort::0::MLMG failed. !!! SIGABRT amrex::Abort::1::MLMG failed. !!! SIGABRT amrex::Abort::2::MLMG failed. !!! SIGABRT amrex::Abort::3::MLMG failed. !!! SIGABRT

I checked my local computer and the plt files seem reasonable for me but I got no ideal why it failed on HPC. Could anyone give any hint on that? Thanks.

[baperry2]

It looks like your simulation might be failing on the MAC projection. To confirm that, follow the troubleshooting instructions here: https://amrex-combustion.github.io/PeleLMeX/manual/html/Troubleshooting.html#linear-solver-failure

What is the difference between the simulation you ran locally and the one you are running on HPC? Did the problem setup or compiler change, or just the number of cores being used?

[potatoslc]

Hi, sorry for the late reply. The only difference is the cores used. The local simulation takes 16 cores and it takes 64 to 128 cores on HPC. Will that be the critical issue?

[baperry2]

Nominally, the code should give identical results as you add cores.