CMEPS atm2ocn and atm2ice weights are unconfigurable

[angus-g]

I'm not sure if this belongs here or at access3-share. I'm trying to set up a global 8km version of ACCESS-OM3 using the grid I generated for Claire and otherwise generally based on the existing 25km RYF configuration. I was using the 2025.05.001 release of the executable.

During initialisation, the MED component calculates regridding weights from the atmosphere to the ocean and ice components:

 (module_med_map: med_map_routehandles_initfrom_field) creating RH patch_uv3d for atm to ocn srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH bilnr for atm to ocn srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consf for atm to ocn srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH patch_uv3d for atm to ice srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH bilnr for atm to ice srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH consf for atm to ice srcMask =    -987987 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH redist for ocn to ice srcMask =          0 dstMask =          0
 (module_med_map: med_map_routehandles_initfrom_field) creating RH redist for ice to ocn srcMask =          0 dstMask =          0

When I ran on about 1200 ocean cores (1664 total), this process was slow-ish, but manageable. But of course, the model itself runs extremely slowly (~3x gridpoints in each direction, plus I haven't even really looked into how to tune things). Scaling up to about 4000 ocean cores, this regridding stage (for the "patch" type) takes hours at initialisation, which seems like a completely unacceptable cost.

I was looking into ways of computing this offline, and noticed that using the cesm coupling mode means the maps between the components are hardcoded.

Now the weird thing is that I thought we configured the datm remapping algorithm in datm.streams.xml, where we're only ever using bilinear remapping. Indeed, unless I'm misunderstanding, at the mediator level, the atmosphere is on the same grid as the ocean, so it shouldn't require any remapping? But perhaps the patch_uv3d transformation is indeed affecting things?

I copied the OM3 executable and patched the string unset to idmap for both the atm2ocn and atm2ice route handles. This should just pass the value straight from datm through to the relevant component, and disables the regridding initialisation yet the model still runs, or at least appears to!

So I guess I have two questions/issues:

1. do we need to be using the patch_uv3d mapping for atmospheric velocities (after already remapping to the model grid), or is the bilinear remapping alone sufficient?;
2. if it is important, should these maps be configurable (e.g. atm2ocn_vmapname which used to exist prior to ESCOMP/CMEPS@e643dfe)

[minghangli-uni]

Thanks @angus-g yes I ran into the same issue when scaling the number of mediator cores up to 384, as shown in the first plot #334 (comment) (red and blue lines). How many cores did you use testing in your config?

I copied the OM3 executable and patched the string unset to idmap for both the atm2ocn and atm2ice route handles. This should just pass the value straight from datm through to the relevant component, and disables the regridding initialisation yet the model still runs, or at least appears to!

Have you compared the outputs when using path_uv3d versus disabling the regridding initialisation? Do they give identical results?

if it is important, should these maps be configurable (e.g. atm2ocn_vmapname which used to exist prior to ESCOMP/CMEPS@e643dfe)

I’m planning to investigate this, thinking of making it configurable, but havent got time to do it yet.

[angus-g]

Ah yeah, I misremembered that the times went up with the number of mediator cores!

How many cores did you use testing in your config?

I was trying with 4448/544, so quite high on the mediator side. I don't have any intuition for what I should be using there, I just assumed it should probably scale up as well given that the domain is finer.

Have you compared the outputs when using path_uv3d versus disabling the regridding initialisation? Do they give identical results?

I have just tried this, and no, they don't reproduce exactly, unfortunately.

Maybe I should try lowering the mediator core count, but I do think it could be valuable to avoid the weight generation in the initialisation of every segment.

[minghangli-uni]

I just assumed it should probably scale up as well given that the domain is finer.

Unfortunately no. The runtime increases almost exponentially as the number of MED cores grows!

but I do think it could be valuable to avoid the weight generation in the initialisation of every segment.

It’s been on my to-do list for a while, I’ll take another look at it.

[anton-seaice]

Thanks @angus-g - ill move to access-om3-configs as its the common spot for these sorts of general issues.

Now the weird thing is that I thought we configured the datm remapping algorithm in datm.streams.xml, where we're only ever using bilinear remapping. Indeed, unless I'm misunderstanding, at the mediator level, the atmosphere is on the same grid as the ocean, so it shouldn't require any remapping?

My understanding is that the remapping in the streams file is remapping between the between the JRA grid ( JRA55do-datm-ESMFmesh.nc ) and the mediator grid (which is the same as the ocean/ice grid, access-om3-25km-ESMFmesh.nc ). Bilinear is used for that step

1. do we need to be using the patch_uv3d mapping for atmospheric velocities (after already remapping to the model grid), or is the bilinear remapping alone sufficient?;

Sorry I don't have an answer, but there are references in the code to vectors being oriented geographically when the fields are passed (e.g. https://github.com/ACCESS-NRI/CICE/blob/6bceb915e232f46a8c84992a3176b98ee0acd8b5/cicecore/drivers/nuopc/cmeps/ice_import_export.F90#L802) , so maybe its related to that ?

I don't think this is something we have changed, it was inherited from CESM. We could reach out to them to ask if needed.

p.s. the patch_uv3d might just be inherited from a coupled model, for OM3, we can probably use nearest but would want to check the coupled fields before changing!

[angus-g]

Thanks @anton-seaice. I was also poking around a bit and saw that it only does this remapping generation when the atm and ocn meshes are different (i.e. mesh_atm and mesh_ocn). So the other question I have is why do we use the "nomask" version for the atm mesh? I saw you mention on the Hive:

There are some cases the nomask version is needed for fields to be coupled conservatively.

But is there a bit more info about this?

[anton-seaice]

In the runoff case , its so that where there are differences in the land mask, any runoff which gets map to land on the mediator grid isn't lost, and can then be remapped into the ocean.

In the atmosphere case, I think the model just used to crash if we didn't set that. I don't recall there being any other better understood reason. @dougiesquire might remember / know

[angus-g]

Makes sense for runoff, but I guess that's also not masked at the component level: model_meshfile and model_maskfile are both the "nomask" variety in drof_in.

There are two places to specify the atmosphere mesh: in mesh_atm in nuopc.runconfig, but also in datm_in. I just tried a quick run changing only the former location (leaving it as "nomask" in datm_in) and it reproduced my run where I explicitly set the idmap for atm2ocn (i.e. no uv3d remapping). I don't know what that proves 😅 but it does help to build up the mental picture of how everything interacts. It still would be nice to have positive documentation on why the specific meshes and masks are chosen though.

[dougiesquire]

In the atmosphere case, I think the model just used to crash if we didn't set that. I don't recall there being any other better understood reason. @dougiesquire might remember / know

Nope, I can't really remember more and my notes have let me down.

Unfortunately I think you are now one of the leading authorities in Aus on how this stuff works @angus-g...

[anton-seaice]

Also @kieranricardo might have thoughts on this question !

[anton-seaice]

where I explicitly set the idmap for atm2ocn (i.e. no uv3d remapping).

Does results from this run differ from the one using uv3d mapping ?

[angus-g]

Does results from this run differ from the one using uv3d mapping ?

Yeah, it does. Although I'm not convinced about which is "correct"? I haven't done any hard analysis, only a month of the 25km. I can look at some differences across fields and see if anything jumps out.

[angus-g]

In the atmosphere case, I think the model just used to crash if we didn't set that.

Can also confirm that the model crashes if model_maskfile and model_meshfile in datm_in are both the masked variant. So that one should probably stay as is! But still possible to change in the nuopc.runconfig at the mediator level.

[dougiesquire]

It's definitely possible that the datm -> ice/ocn remappings are not necessary and are just leftovers from the CESM set-up we started with. Even if this is the case, it's not surprising I suppose that turning it off changes answers.