Add support for Pawsey Setonix

[ScottWales]

Fixes #16

[hiker]

@ScottWales , we just had our FAB meeting, and Deva from Pawsey pointed out the following instructions for using cce 18:
https://pawsey.atlassian.net/wiki/spaces/US/pages/695074817/March+2025+Software+Update+-+Important+Information
Not sure if you are already aware of it, I didn't check what's in that document :)

Apparently, cce16 is an absolute nightmare :)

[ScottWales]

Thanks for the info - at the moment this is set up to use cray 16 as with 18 I was getting linker errors when compiling with mpif90 - it was linking in the wrong version of the fortran runtime. I've just about got it all working now - the spack environment is fully compiled, I can look again at updating to 18 once I have the lfric test completed (still need to fix some environment variables)

[hiker]

I forgot to add the link: https://github.com/MetOffice/NG-ARCH-MBN/issues/18#issuecomment-2735625087
Can you access this? It contains the version numbers that Deva used, since he had some problems with certain versions.

[ScottWales]

I don't have access to that repo

[hiker]

I'll copy the interesting looking comments:
1.

The new software stack using cpe/24.07 is live on Setonix. It provides cce18 and gcc-native/13.2 (13.3.0). Chris suggested to install the software dependencies for cce18, which supports OpenACC offloading.

I'm using the recipes from https://github.com/MetOffice/simit-spack

and having issues compling gftl-shared library in Cray env required to build pfunit:

8 CMake Error at CMakeLists.txt:35 (include):
9 include could not find requested file:
10
11 Cray

There is no Cray support in the package. Currently working on creating the required files in the package to compile for Cray env.

apr@1.6.2 required to compile perl-alien-svn (dependency of fcm) fails to compile in Cray env but > succeeds in GNU:

437 checking which format to use for apr_ssize_t... %ld
438 checking size of size_t... 0
439 checking which format to use for apr_size_t... %lu
440 checking size of off_t... 0
441 checking which type to use for apr_off_t... configure: error: could not determine the size of > off_t

See build log for details:
/scratch/pawsey0001/ddeeptimahanti/setonix/2025.03/software/ddeeptimahanti/build_stage/> spack-stage-apr-1.6.2-yhykfnvt22idrf3alcx2qgujhgabx4jh/spack-build-out.txt

So, installing fcm with gcc-native/13.2

Compiled py-psyclone/3.1.0, py-jinja2/3.1.4, perl-config-inifiles/3.000003 using cce/18.0.0

Compiled fcm2021.05.0 in GCC due to apr/1.6.2 failure using cce/18.0.0
Still working on pfunit/4.11.1 and gftl-shared using cce/18. Currently, there is no Cray support in
the source code.
Compiled blitz in Cray env without papi support. Blitz hasn't been updated since 2019 and throws
undeclared identifier errors when using the papi/7.0.1.2 on Setonix

Yaxt is failing with undefined reference errors:

2687 The Cpu Module:(.text+0x21c5): undefined reference to `xt_xmap_dis
t_dir_intercomm_new_f'
2688 make[1]: *** [Makefile:1809: test_xmap_dist_dir_intercomm_parallel
_f] Error 1
Issues compiling Xios/2252 with Spack. For the time being looking to install it manually like > before.

Emailed Andy and Sam to know the status of these packages in Cray env.

[ScottWales]

The environment is working on pawsey, to use:

    module use "/scratch/pawsey0835/swales/ngmo-envs/modules"
    module load "lfric/setonix"

then put envrun before any compile or run commands (envrun bash will start a shell in the container environment)

[ScottWales]

Issues

• Spack lib path not being searched for -lxios, -lyaxt
• Cray netcdf lib path not being searched for -lnetcdff (because lfric is getting linked with mpifort rather than ftn)
• LD_LIBRARY_PATH is not set
• mpirun is not on PATH so MPI example doesn't work