Troubles with check_maps gmake andor instructions

[kdraeder]

I'm trying to make new mapping+domain files and am stuck on the check_maps build stage.
I'm trying to follow the instructions in
https://github.com/ESMCI/cime/blob/master/tools/mapping/check_maps/README.
I've loaded one of derecho's current esmf modules: esmf/8.8.1.
I can't find an equivalent of esmf-7.0.0-ncdfio-uni-O for 8.8.1.

module load esmf-8.8.1-ncdfio-uni-O
Lmod has detected the following error:
The following module(s) are unknown: "esmf-8.8.1-ncdfio-uni-O"
module avail ncdfio
No module(s) or extension(s) found!

If I assume that that module is no longer needed(?)
and proceed to the gmake step I get "gmake: No match."
I've added '-d --debug' to the gmake, but get no more useful information.
The Makefile is there:
-rw-r--r-- 1 raeder p86850054 2657 Sep 11 16:54 ../check_maps/src/Makefile
so I don't know what gmake is trying to match. The ncdfio module?

Any suggestions?
Thanks,
Kevin

[billsacks]

Hmmm, yeah, this tool has probably fallen into disuse/mostly unsupported territory, since we're no longer making many mapping files. I just tried it myself and got a different error:

/opt/cray/pe/mpich/8.1.29/ofi/intel/2022.1/bin/mpif90 -cxxlib -Wl,--no-as-needed  -qopenmp -L/glade/u/apps/derecho/24.12/spack/opt/spack/esmf/8.8.1/cray-mpich/8.1.29/oneapi/2024.2.1/oigq/lib -L/glade/u/apps/casper/24.12/spack/opt/spack/intel-oneapi-mkl/2024.2.2/oneapi/2024.2.1/kvnq/lib -L/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf-c/4.9.3/cray-mpich/8.1.29/oneapi/2024.2.1/bvx7/lib64 -L/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.29/oneapi/2024.2.1/fgac/lib -L/glade/u/apps/derecho/24.12/spack/opt/spack/parallel-netcdf/1.14.0/cray-mpich/8.1.29/oneapi/2024.2.1/la34/lib -L/glade/u/apps/derecho/24.12/spack/opt/spack/parallelio/2.6.6/cray-mpich/8.1.29/oneapi/2024.2.1/5oef/lib  \
  -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/esmf/8.8.1/cray-mpich/8.1.29/oneapi/2024.2.1/oigq/lib  -Wl,-rpath,/glade/u/apps/casper/24.12/spack/opt/spack/intel-oneapi-mkl/2024.2.2/oneapi/2024.2.1/kvnq/lib -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf-c/4.9.3/cray-mpich/8.1.29/oneapi/2024.2.1/bvx7/lib64 -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf-fortran/4.6.1/cray-mpich/8.1.29/oneapi/2024.2.1/fgac/lib  -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/parallel-netcdf/1.14.0/cray-mpich/8.1.29/oneapi/2024.2.1/la34/lib  -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/parallelio/2.6.6/cray-mpich/8.1.29/oneapi/2024.2.1/5oef/lib -o ../ESMF_RegridWeightGenCheck ESMF_RegridWeightGenCheck.o -lesmf -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpnetcdf -lpioc  -lrt -ldl
ifort: remark #10448: Intel(R) Fortran Compiler Classic (ifort) is now deprecated and will be discontinued late 2024. Intel recommends that customers transition now to using the LLVM-based Intel(R) Fortran Compiler (ifx) for continued Windows* and Linux* support, new language support, new language features, and optimizations. Use '-diag-disable=10448' to disable this message.
ld: cannot find -lmkl_intel_lp64
ld: cannot find -lmkl_sequential
ld: cannot find -lmkl_core
make: *** [Makefile:53: ../ESMF_RegridWeightGenCheck] Error 1

This is with the following modules:

  1) ncarenv/24.12 (S)   2) craype/2.7.31   3) nco/5.3.1   4) ncview/2.1.9   5) intel-oneapi/2024.2.1   6) ncarcompilers/1.0.0   7) libfabric/1.15.2.0   8) cray-mpich/8.1.29   9) hdf5/1.12.3  10) netcdf/4.9.2  11) esmf/8.8.1

In my case, I think the issue is that there are link paths (e.g., for mkl) pointed to in the ESMFMKFILE that no longer exist. Presumably they existed when the ESMF module was created, but paths changed since then.

Maybe a solution (for now) is to ask the cisl help desk to install a new ESMF module for the latest ESMF release (8.9.0), which would presumably have the correct paths since it's new. @kdraeder does that seem like a reasonable next step to you? It seems good to get that installed anyway, so I can get that rolling.... I'm not sure if this will solve your issue, but it seems like we should solve this first and then try to figure out any remaining errors.

And once we get this working, we should update the documentation....

[kdraeder]

@billsacks thanks for your quick test and response!
Is this software not used because all of the files required by supported grids have been created,
or because there's replacement software somewhere?

My impression is that this needs to be run as serial programs, but your example is using mpif90.
Maybe that's being told to build a serial program.

I also see the note that ifort was (?) discontinued in late 2024. Does that mean that derecho
has an unsupported compiler?

I'll follow your lead on the new ESMF module. It seems strange that the current versions might be broken
but I'm the first person to stumble on it. Especially for such a widely used module.

Beyond that, it appears that your build is actually getting into gmake and doing something,
but mine doesn't seem to reach an mpif90 command, so I seem to have an additional problem
(likely a user error, but I looked at everything I could think of).

[billsacks]

Is this software not used because all of the files required by supported grids have been created,
or because there's replacement software somewhere?

This tool is used much less now for CESM because most of the mapping weights are created at runtime rather than in a preprocessing step. In investigating this, I did find that a subset of the functionality of this tool is available if you specify the --check flag to ESMF_RegridWeightGen, but the official ESMF version of the tool is missing some of the functionality of the CIME version (see esmf-org/esmf#496).

My impression is that this needs to be run as serial programs, but your example is using mpif90.
Maybe that's being told to build a serial program.

Hmmm, good point. I just retried with these modules:

  1) ncarenv/24.12 (S)   2) craype/2.7.31   3) intel/2025.1.0   4) ncarcompilers/1.0.0   5) hdf5/1.12.3   6) mpi-serial/2.5.0   7) esmf/8.8.1

but got the same issue I ran into before:

/glade/u/apps/casper/24.12/spack/opt/spack/intel-oneapi-compilers/2025.1.0/gcc/12.4.0/47hn/compiler/latest/bin/ifx -c -O -fPIC -debug minimal -assume realloc_lhs -m64 -mcmodel=small -pthread -threads  -qopenmp -I/glade/u/apps/derecho/24.12/spack/opt/spack/esmf/8.8.1/oneapi/2025.1.0/u5vi/include -I/glade/u/apps/derecho/24.12/spack/opt/spack/esmf/8.8.1/oneapi/2025.1.0/u5vi/include -I/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf/4.9.3/packages/netcdf-c/4.9.3/oneapi/2025.1.0/atex/include -I/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf/4.9.3/packages/netcdf-fortran/4.6.1/oneapi/2025.1.0/5ksy/include -I/glade/u/apps/derecho/24.12/spack/opt/spack/parallelio/2.6.6/oneapi/2025.1.0/nlfy/include \
   -DESMF_NO_INTEGER_1_BYTE -DESMF_NO_INTEGER_2_BYTE -DESMF_MPIUNI -I/glade/derecho/scratch/csgteam/temp/spack/derecho/24.12/builds/spack-stage-esmf-8.8.1-u5vi6hmg3yyyh5tjmznrhexdp5jwphl7/spack-src/src/Infrastructure/stubs/mpiuni -DESMF_MOAB=1 -DESMF_LAPACK=1 -DESMF_NO_ACC_SOFTWARE_STACK=1 -DESMF_NETCDF=1 -DESMF_YAMLCPP=1 -DESMF_YAML=1 -DESMF_PIO=1 -DESMF_NO_OPENACC -DESMF_BOPT_O -DESMF_TESTPERFORMANCE -DESMF_TESTCOMPTUNNEL -DSx86_64_small=1 -DESMF_OS_Linux=1 -DESMF_COMM=mpiuni -DESMF_DIR=/glade/derecho/scratch/csgteam/temp/spack/derecho/24.12/builds/spack-stage-esmf-8.8.1-u5vi6hmg3yyyh5tjmznrhexdp5jwphl7/spack-src ESMF_RegridWeightGenCheck.F90
/glade/u/apps/casper/24.12/spack/opt/spack/intel-oneapi-compilers/2025.1.0/gcc/12.4.0/47hn/compiler/latest/bin/ifx -m64 -mcmodel=small -pthread -threads -cxxlib -Wl,--no-as-needed  -qopenmp -L/glade/u/apps/derecho/24.12/spack/opt/spack/esmf/8.8.1/oneapi/2025.1.0/u5vi/lib -L/glade/u/apps/casper/24.12/spack/opt/spack/intel-oneapi-mkl/2025.1.0/oneapi/2025.1.0/f76g/lib -L/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf/4.9.3/packages/netcdf-c/4.9.3/oneapi/2025.1.0/atex/lib64 -L/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf/4.9.3/packages/netcdf-fortran/4.6.1/oneapi/2025.1.0/5ksy/lib -L/glade/u/apps/derecho/24.12/spack/opt/spack/parallelio/2.6.6/oneapi/2025.1.0/nlfy/lib  \
  -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/esmf/8.8.1/oneapi/2025.1.0/u5vi/lib  -Wl,-rpath,/glade/u/apps/casper/24.12/spack/opt/spack/intel-oneapi-mkl/2025.1.0/oneapi/2025.1.0/f76g/lib -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf/4.9.3/packages/netcdf-c/4.9.3/oneapi/2025.1.0/atex/lib64 -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/netcdf/4.9.3/packages/netcdf-fortran/4.6.1/oneapi/2025.1.0/5ksy/lib  -Wl,-rpath,/glade/u/apps/derecho/24.12/spack/opt/spack/parallelio/2.6.6/oneapi/2025.1.0/nlfy/lib -o ../ESMF_RegridWeightGenCheck ESMF_RegridWeightGenCheck.o -lesmf -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -lrt -ldl
ld: cannot find -lmkl_intel_lp64
ld: cannot find -lmkl_sequential
ld: cannot find -lmkl_core
gmake: *** [Makefile:53: ../ESMF_RegridWeightGenCheck] Error 1

I'll follow your lead on the new ESMF module. It seems strange that the current versions might be broken
but I'm the first person to stumble on it. Especially for such a widely used module.

I don't think the ESMF modules are necessarily broken for most usages. The issue seems to be that the makefile for this tool just pulls information from the ESMFMKFILE, and for some reason that file has gotten out-of-date with respect to the current directory organization (from what I can tell). So if we had a more robust build that didn't rely on asking the ESMFMKFILE where to find various libraries but instead was more self-contained / self-reliant, we probably wouldn't be running into this problem.

Beyond that, it appears that your build is actually getting into gmake and doing something,
but mine doesn't seem to reach an mpif90 command, so I seem to have an additional problem
(likely a user error, but I looked at everything I could think of).

Yeah, I'm not sure what to make of that. Let me know if you'd like to sit down together - in person or on a call - to troubleshoot.

[kdraeder]

@billsacks Thanks for all of the answers.
How expensive is the runtime creation of mapping weights?
It probably doesn't matter for climate runs, but I'm trying to use a 1/4 degree data ocean
with cam-se (ne30 and regionally refined) in data assimilation,
so I have many short forecasts and need to minimize start costs.

With fresh eyes this morning I saw that I was taking the check_maps instructions too literally
and had [ ] around the VERBOSE=TRUE. Removing those and running gmake with the debug args yielded
the same problem you reported; ld couldn't find several libraries.

[billsacks]

I don't know off-hand how expensive runtime creation of mapping weights is. We started discussing this in ESCOMP/CMEPS#335, but haven't prioritized that because - as far as I know - the generation of mapping weights hasn't been a bottleneck for most use cases. I understand that your use case stresses the system in new ways, so we may need to prioritize that.

I tried building with the newly-installed esmf 8.9.0, but ran into the same issue. I have passed that along to Brian V in CISL to see if he has any insight. However, I was able to build this tool with gcc instead of intel - so with these modules:

  1) ncarenv/24.12 (S)   2) craype/2.7.31   3) nco/5.3.1   4) ncview/2.1.9   5) gcc/12.4.0   6) ncarcompilers/1.0.0   7) hdf5/1.12.3   8) netcdf/4.9.2   9) mpi-serial/2.5.0  10) esmf/8.9.0

(I haven't tried actually running the resulting tool.)