Crashes due to MPAS-Ocean state validation do not stop E3SM or standalone MPAS-Ocean runs

[xylar]

For a long time, we have been aware that state validation leads to hanging jobs in standalone MPAS-Ocean runs. I have wanted to look into this but haven't found the time. I just saw the same behavior in my E3SM run at:

/lcrc/group/e3sm/ac.xylar/scratch/chrys/20250618.GMPAS-JRA1p5-DIB-PISMF.TL319_SOwISC12to30E3r4.upwind_factor.chrysalis/run

Although the job crashes almost immediately, the allocation is not cancelled:

            764638     debug run.2025 ac.xasay  R    2:48:39      4 chr-[0495-0498] 
            764661     debug run.2025 ac.xasay  R    1:18:41      4 chr-[0490-0493] 

This happened twice in succession, leading to 2 jobs hanging and blocking scarce resources.

[xylar]

Error files are clearly visible, e.g.:

$ tail log.ocean.0056.err 
----------------------------------------------------------------------
Beginning MPAS-ocean Error Log File for task      56 of     128
    Opened at 2025/06/18 10:22:49
----------------------------------------------------------------------

CRITICAL ERROR: ERROR: State validation failed. See block stats files for more information.
Logging complete.  Closing file at 2025/06/18 10:22:49

[jonbob]

@xylar -- I had looked at this once and have some ideas on how to solve it. I'll try to dig in, or we can talk about it

[xylar]

Sure, no rush. I just figured it's well past time that we had an issue for it.

[jonbob]

@xylar -- can you please tell me how to replicate this case? I have something I'd like to test

[xylar]

I'm sure there's an easier way but, starting from #7371, I set up a GMPAS-JRA1p5-DIB-PISMF.TL319_SOwISC12to30E3r4.chrysalis case with the following custom namelist options:

config_thickness_flux_type = 'upwind'
config_use_spatially_variable_upwind = .true.
config_spatially_variable_upwind_hmin = 10.0
config_spatially_variable_upwind_hmax = 20.0

The crash happens a few time steps into the test so the default 5-day smoke test is plenty to reproduce.