PIO error when running a hybrid run (E3SMv2.1)

[AlexAudette]

I am trying to get an ensemble of simulations all starting from the same restarts. To do this, I am creating a hybrid case and using the pertlim namelist parameter. The run appear to initialize properly, but before the start of the first time step, I get a PIO error that seems to stem from a wrong file format issue in the low level I/O module. I copied the relevant log below.

The issue appears to be similar to #6195, but this is using maint-2.1, not 2.0.

Thanks in advance for your help, and please let me know if you need more information.

Some additional tests I performed:

• I was able to get a branch run running properly (though the pertlim parameter doesn't work for branch runs).
• Setting pertlim to 0.0 does not change anything.

Relevant log:

0: PIO: ERROR: Opening file (PI_Control_wtrc_r2.mpaso.rst.am.timeSeriesStatsMonthly.0001-01-01_00000.nc) with iotype 2 (PIO_IOTYPE_NETCDF) failed. The low level I/O library call failed. NetCDF: Unknown file format (error num=-51), (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pioc_support.c:3916)
0: PIO: ERROR: Opening file (PI_Control_wtrc_r2.mpaso.rst.am.timeSeriesStatsMonthly.0001-01-01_00000.nc) failed. Internal error. NetCDF: Unknown file format (error num=-51), (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pioc_support.c:4022)
0: PIO: ERROR: Opening file (PI_Control_wtrc_r2.mpaso.rst.am.timeSeriesStatsMonthlyMax.0001-01-01_00000.nc) with iotype 2 (PIO_IOTYPE_NETCDF) failed. The low level I/O library call failed. NetCDF: Unknown file format (error num=-51), (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pioc_support.c:3916)
0: PIO: ERROR: Opening file (PI_Control_wtrc_r2.mpaso.rst.am.timeSeriesStatsMonthlyMax.0001-01-01_00000.nc) failed. Internal error. NetCDF: Unknown file format (error num=-51), (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pioc_support.c:4022)
0: PIO: ERROR: Opening file (PI_Control_wtrc_r2.mpaso.rst.am.timeSeriesStatsMonthlyMin.0001-01-01_00000.nc) with iotype 2 (PIO_IOTYPE_NETCDF) failed. The low level I/O library call failed. NetCDF: Unknown file format (error num=-51), (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pioc_support.c:3916)
0: PIO: ERROR: Opening file (PI_Control_wtrc_r2.mpaso.rst.am.timeSeriesStatsMonthlyMin.0001-01-01_00000.nc) failed. Internal error. NetCDF: Unknown file format (error num=-51), (/global/u2/a/aaudette/E3SM/externals/scorpio/src/clib/pioc_support.c:4022)
459: MPICH ERROR [Rank 459] [job id 32305791.0] [Tue Oct 29 11:13:32 2024] [nid005415] - Abort(538976288) (rank 459 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 459
459:
459: aborting job:
459: application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 459

[ MPICH ERROR repeating until the end of log]

[jonbob]

Hi Alex -- my name is Jon Wolfe and I work on E3SM at LANL on the mpas components.

Those files are usually missing from a hybrid run, and they should not cause a failure. Is there more in the log file? My guess is that you will need to modify the mpaso and mpassi restart files to remove (or rename) a variable called xtime before you can use them for a hybrid run. There is an ncl command that does that easily -- it looks like:

ncrename -v xtime,xtime.orig {initial condition file}

Otherwise the mpas framework is likely complaining that it can't find the correct xtime in those files.

Please let me know if that helps,
Jon

[AlexAudette]

Wonderful! It seems to be working without any issue at the moment. Is there any further documentation about this that I could go read to better understand the issue or hybrid runs in E3SM?

[jonbob]

I don't know how well it's documented. I can help explain the issue, which is mostly that the mpas framework thinks an initial file is applicable only for the xtime value if there is one. It's why a branch run would work -- you're not changing the model start date -- but a hybrid will typically fail. But please let me know know if you'd like more information

[AlexAudette]

Thanks for the explanation and your help!