EAMxx: Data interpolation for MAM4 microphysics. #7501

odiazib · 2025-07-10T00:26:15Z

Our current implementation of the reader for Linoz, invariants, and vertical emission files uses an approach based on the former SPA reader. @bartgol implemented the DataInterpolation class, which provides more flexibility to handle different types of files. Here, we are using this DataInterpolation class to read the necessary files for the MAM4 microphysics interface. Because MAM4 still uses the vertical interpolation from EAM (the ported version), I have added three new vertical interpolation types to this class and created a VerticalRemapperMAM4 class for MAM4 vertical interpolation.

This PR also fixes the time interpolation issue for Linoz, invariant, and vertical emission fields.

Because we will use single-year NetCDF files for elevated emissions, invariants, and Linoz, the following namelist parameters were deleted: mam4_linoz_ymd, mam4_oxid_ymd, and elevated_emiss_ymd.

I implemented the VerticalRemapperMAM4 class, which inherits from VerticalRemapper, to invoke MAM4xx vertical interpolation routines that were ported from EAM. The DataInterpolation class handles VerticalRemapperMAM4 as a custom vertical remapper.

In particular, VerticalRemapperMAM4 supports three types of vertical interpolation:

1. `VerticalRemapperMAM4::MAM4_PSRef` This is equivalent to `DataInterpolation::VRemapType::Dynamic3DRef`, where the source pressure field is computed as:

psrc(icol,k)=ps_v(icol)⋅hybm(k)+P0⋅hyam(k)

The main difference is that MAM4_PSRef invokes the mam4::vertical_interpolation::vert_interp routine.

I added a boolean parameter, mam4_use_mam4xx_oxi_vert_remap, to allow users to switch between MAM4_PSRef and DataInterpolation::VRemapType::Dynamic3DRef. Currently, this parameter is hardcoded to true, meaning oxidant vertical interpolation will be performed using the MAM4xx routine. However, if the evaluation team is interested in testing the use of DataInterpolation::VRemapType::Dynamic3DRef, this parameter can be moved to the namelist.

VerticalRemapperMAM4::MAM4_ZONAL

This vertical interpolation type is similar to DataInterpolation::Static1D, where a 1D variable is used as the independent variable. In the case of MAM4_ZONAL, the independent variable is lev with units of mbar. The main difference is that MAM4_ZONAL invokes the mam4::vertical_interpolation::vert_interp routine. Note that in this case, unit conversion from mbar to Pascals is handled by mam4::vertical_interpolation::vert_interp.

3. VerticalRemapperMAM4::MAM4_ELEVATED_EMISSIONS
This vertical interpolation type uses altitude as the independent variable. There is no equivalent in the VerticalRemapper for this case.

Tests (DataInterpolation for elevated emissions)

I tested this PR using only the DataInterpolation for elevated emissions. In this case, we expect the tests to pass, including baselines (BFB). However, I noted that the single-precision tests fail for gcc-openmp and gcc-cuda during the baseline comparison. Additionally, the test REP_D_Ln5.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.ghci-snl-cpu_gnu is also failing in the baseline comparison.

gcc-cuda/sp and gcc-openmp/sp

Click to view detailed comparison

Most of the standalone tests are passing, except for gcc-cuda/sp and gcc-openmp/sp. It is important to note that the double-precision tests are passing, which means this implementation using the DataInterpolation class is producing BFB results consistent with the former reader (using @TaufiqHassan's correction for time interpolation).

REP_D_Ln5.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.ghci-snl-cpu_gnu

Click to view detailed comparison

This test is failing with DIFFs, and these DIFFs increase as the number of time steps increases. I carefully reviewed this test and concluded that these DIFFs are very small in the first iteration. However, as MAM4 affects host model variables (in particular, temperature), these DIFFs increase over time. I conducted a couple of experiments where I ran both the old and new readers, and I found that the DIFFs remain small. For example:

Sector	BFB Hash	Value
forc_new_8_sector_0	12e765df7450c4b7	243788.7310767502
forc_old_8_sector_0	12e765df7450e328	243788.7310767502
forc_new_8_sector_1	4805cb37e8068d23	151464.7675245383
forc_old_8_sector_1	4805cb37e80691e7	151464.7675245383
forc_new_8_sector_2	3e98c5625d38c1fd	262341.6782332305
forc_old_8_sector_2	3e98c5625d38caa5	262341.6782332306

Using preprocessor directives (USE_OLD_LINOZ_FILE_READ and USE_OLD_VERTICAL_FILE_READ) was causing confusion for the evaluation team. Therefore, I decided to delete the former reader.

Tests (DataInterpolation for both invariants and linoz)

Before this PR, we knew there was a time interpolation issue inherited from the former SPA reader. Therefore, using the DataInterpolation class for invariants and Linoz will produce NBFB results in all tests where microphysics processes are involved. Thus, baseline tests are expected to fail for gcc-cuda, gcc-openmp, and REP_D_Ln5.ne4pg2_oQU480.F2010-EAMxx-MAM4xx.ghci-snl-cpu_gnu.

During the testing of DataInterpolation for invariants and Linoz, we found a bug in the former reader and noted that the DataInterpolation class only works for files containing single-year data. The bug in the former reader involved selecting the wrong year for interpolation during a restart, which caused the restart to be NBFB. This bug was fixed by replacing the NetCDF files for oxidants and Linoz. The new files contain only single-year data to accommodate the current features of the DataInterpolation class.

NC files

Elevated emissions

The DataInterpolation class relies on the time variable for time interpolation. In the elevated emission files currently used in the master branch, the time variable is not present. Instead, these files have the date variable. @meng630 helped add the time variable to these files. Note that we do not expect any changes due to these file updates.

Invariants

We are using new invariant files for the year 2015. We note that the current files do not contain data from 2010.

Linoz

We are using a single-year file for Linoz for the year 2010. In addition, we replaced the time variable and created a new time variable that does not assume years of 365 days and months of 30 days, which was causing interpolation issues in the DataInterpolation class. Note that the DataInterpolation class uses time instead of date for time interpolation.

bartgol

I am not entirely sure if this should be an atm proc, or if you should put the data read/interpolation feature inside the already existing eamxx_mam_generic_process_interface.*pp files.

That said, I have some thoughts, and some cleanup suggestions. I did not get into the details of the mam processes or in the reader utils.

bartgol · 2025-07-10T20:26:41Z

components/eamxx/cime_config/namelist_defaults_eamxx.xml

+      <!-- LINOZ parameters -->
+      <mam4_linoz_ymd type="integer" > 20100101</mam4_linoz_ymd>
+      <mam4_linoz_file_name type="file" doc="LINOZ chemistry file"> ${DIN_LOC_ROOT}/atm/scream/mam4xx/linoz/ne30pg2/linoz1850-2015_2010JPL_CMIP6_10deg_58km_ne30pg2_c20240724.nc</mam4_linoz_file_name>
+      <mam4_linoz_file_name hgrid="ne4np4.pg2" type="file" doc="LINOZ chemistry file"> ${DIN_LOC_ROOT}/atm/scream/mam4xx/linoz/ne4pg2/linoz1850-2015_2010JPL_CMIP6_10deg_58km_ne4pg2_c20240724.nc</mam4_linoz_file_name>


This is fine, but FYI: if you have multiple copies of a parameter (like mam4_linoz_file_name here), the metadata type and doc only need to be set once in the first occurrence. It's ok to put it in all of them, but for long doc strings, it may help to just add it once.

Again, this is fine.

Since I plan to delete the new process mam4_interpolation_aero_microphys, all duplicate parameters will be removed. I will move the reader to the aero_microphys atm process.

bartgol · 2025-07-10T20:37:00Z

components/eamxx/src/physics/mam/eamxx_mam_interpolation_microphysics_process_interface.cpp

+  const auto &wet_atm = wet_atm_;
+  const auto &dry_atm = dry_atm_;
+
+  const auto scan_policy = ekat::ExeSpaceUtils<


A few thoughts on this:

You only need to compute vertical layer heights if you have an elevation-based interpolation no? Wouldn't it make sense to put the calc of zmid in an if statement?

There seem to be quite a bit of code related to computing this quantity. Wild idea: why not storing a VerticalLayerDiagnostic obj in this class, and running the diag to compute the zmid field? Wouldn't you be able to get rid of a bunch of code in this class (e.g., buffers, and maybe even wet/dry stuff)?

Yes, it does. I copied and pasted this block of code from the microphysics interface, where zmid is needed. I will delete this interface.

The wet/dry block of code is from the microphysics interface. I will explore the VerticalLayerDiagnostic approach.

@bartgol Can we review the VerticalLayerDiagnostic in a follow-up pull request (PR)?

bartgol · 2025-07-10T20:38:12Z

components/eamxx/src/physics/mam/eamxx_mam_interpolation_microphysics_process_interface.hpp

+// The process responsible for handling MAM4 aerosol microphysics. The AD
+// stores exactly ONE instance of this class in its list of subcomponents.
+class MAMInterpolationMicrophysics final : public MAMGenericInterface {
+  using PF = scream::PhysicsFunctions<DefaultDevice>;


There's a few unused typedefs here. Let's keep the class header lean (and easier to read).

This interface will be removed.

bartgol · 2025-07-10T20:40:02Z

components/eamxx/src/physics/mam/readfiles/vertical_remapper_mam4.cpp

+{
+  m_src_pmid=p;
+}
+void VerticalRemapperMAM4::


I think you can get rid of the set_xyz_pressure impl, since the base class impl should now be enough.

I removed set_source_pressure(const Field& p). For the elevated emission case, which is Custom type in DataInterpolation, I am still using set_target_pressure(const Field& p) and set_source_pressure(const std::string& file_name).

bartgol · 2025-07-10T20:44:06Z

components/eamxx/src/physics/mam/readfiles/vertical_remapper_mam4.hpp

+
+protected:
+
+  using KT = KokkosTypes<DefaultDevice>;


These typedefs are also unused and can be removed. Even ThreadTeam is not strictly needed, since you can just use auto in the lambda (but at least you could mv the using line to the cpp, where it's used, so you can rm the mam4.hpp include, and keep the header leaner).

components/eamxx/tests/multi-process/physics_only/shoc_cld_p3_rrtmgp/input.yaml

singhbalwinder

These comments are likely for old version of this PR (likely on the deleted files for which the code has been moved). Ignore if not relevant.

singhbalwinder

Thanks, Oscar for taking on this major task! I have left some minor comments and suggestions. Overall looks great!

Could you please add some comments to this code? It would be good to add comments to at least separate out sections for different types of files (e.g., oxidants, LINOZ, elevated emissions).
Also adding comments about what kind of interpolation they need (time, horizontal, vertical...and what kind of vertical). It will also be good to mention the zonal LINOZ file (lat,lev) in the comments here.

components/eamxx/src/physics/mam/eamxx_mam_microphysics_process_interface.cpp

singhbalwinder · 2025-07-17T00:01:15Z

components/eamxx/src/physics/mam/eamxx_mam_microphysics_process_interface.cpp

+  const auto oxid_O3 = get_field_out("O3").get_view<Real **>();
+  const auto oxid_OH = get_field_out("OH").get_view<Real **>();
+  const auto oxid_NO3 = get_field_out("NO3").get_view<Real **>();
+  const auto oxid_HO2 = get_field_out("HO2").get_view<Real **>();


We already have all these names (linoz and oxids) in a vector. Can we use a for-loop instead of explicitly using names?

components/eamxx/src/physics/mam/readfiles/vertical_remapper_mam4.cpp

components/eamxx/src/physics/mam/eamxx_mam_microphysics_process_interface.cpp

components/eamxx/cime_config/namelist_defaults_eamxx.xml

TaufiqHassan

Hi @odiazib, this looks great! I've looked at the external forcings from a ne4pg2 simulation and compared the results against an older simulation using PR#7381

The results look consistent as expected from the description. Testing it on ne30pg2 grid against EAM.

TaufiqHassan

The ne30pg2 comparison against EAM looks consistent with different 2016 emissions data.

…tion based readers are part of the microphysics interface. EAMxx: Deleted old header.

EAMxx: Fixing compilation error in CUDA.

EAMxx: Fixing failing tests.

EAMxx: Using z_mam4_int.

… and elevated_emiss_ymd from namelist.

odiazib requested a review from bartgol July 10, 2025 14:44

odiazib marked this pull request as ready for review July 10, 2025 16:20

odiazib requested a review from singhbalwinder July 10, 2025 16:20

kaizhangpnl added the MAM4xx MAM4xx related changes label Jul 10, 2025

bartgol approved these changes Jul 10, 2025

View reviewed changes

odiazib changed the title ~~EAMxx: Data interpolation for Linoz and oxid in MAM microphysics.~~ EAMxx: Data interpolation for MAM4 microphysics. Jul 11, 2025

odiazib marked this pull request as draft July 11, 2025 23:10

odiazib force-pushed the eamxx/data_interpolation_microphysics branch from 8f298b9 to 67f32dc Compare July 12, 2025 20:59

odiazib marked this pull request as ready for review July 14, 2025 23:06

odiazib requested a review from bartgol July 14, 2025 23:10

bartgol reviewed Jul 15, 2025

View reviewed changes

components/eamxx/tests/multi-process/physics_only/shoc_cld_p3_rrtmgp/input.yaml Outdated Show resolved Hide resolved

odiazib force-pushed the eamxx/data_interpolation_microphysics branch 3 times, most recently from 27b4ad6 to 52c6ebe Compare July 16, 2025 15:41

odiazib mentioned this pull request Jul 16, 2025

Updates for PR 7501 in E3SM eagles-project/mam4xx#465

Merged

singhbalwinder reviewed Jul 16, 2025

View reviewed changes

singhbalwinder reviewed Jul 17, 2025

View reviewed changes

odiazib force-pushed the eamxx/data_interpolation_microphysics branch 2 times, most recently from b16a666 to 6d8919a Compare July 21, 2025 23:20

rljacob assigned bartgol Jul 24, 2025

odiazib force-pushed the eamxx/data_interpolation_microphysics branch from 5a8a484 to 913e47c Compare July 24, 2025 17:43

odiazib added the non-BFB PR makes roundoff changes to answers. label Jul 28, 2025

odiazib force-pushed the eamxx/data_interpolation_microphysics branch from 913e47c to 4b5cdf0 Compare July 28, 2025 15:56

odiazib requested review from singhbalwinder, TaufiqHassan and mahf708 July 28, 2025 16:09

TaufiqHassan reviewed Jul 28, 2025

View reviewed changes

components/eamxx/cime_config/namelist_defaults_eamxx.xml Show resolved Hide resolved

TaufiqHassan reviewed Jul 28, 2025

View reviewed changes

TaufiqHassan self-requested a review July 30, 2025 15:52

TaufiqHassan approved these changes Jul 30, 2025

View reviewed changes

odiazib mentioned this pull request Sep 2, 2025

EAMxx: Adding a time offset index during the initialization of Linoz and oxidant files. #7654

Merged

odiazib added 27 commits September 15, 2025 15:17

EAMxx: vertical remapper for MAM4xx.

bc3ae57

EAMxx: atm process interface for mam interpolation.

3974738

EAMxx: single and multi-process test for interpolation atm-process.

71b9088

EAMxx: namelist for interpolation atm-process.

53394fe

EAMxx: Using precompiled directives to turn off/on the former reader.

17fb5ab

EAMxx: Using FM to store z_iface.

2f9cd11

EAMxx: Using the DataInterpolation reader in the microphysics interface.

c059ab5

EAMxx: Using new reader for vertical emissions.

f2d816a

EAMxx: Removing the interpolation microphysics process. DataInterpola…

53fde21

…tion based readers are part of the microphysics interface. EAMxx: Deleted old header.

EAMxx: Adding z_iface : 0.0 to yaml file in standalone tests.

f14eb51

EAMxx: Adding z_iface to namelist.

0ba823c

EAMx: Fixing dynamics_physics test for microphysics.

4bbb366

EAMxx: Removing unused code.

396d392

EAMxx: Fixing compilation error in CUDA.

EAMxx: Do not run test on AT2.

592f7b1

EAMxx: Fixing variable names and add comments.

03d019e

EAMxx: Fixing after rebasing against master.

4c965ca

EAMxx: Fixing failing tests.

EAMxx: Using (1,1,1) as ref_ts.

a44fefc

EAMxx: Rename z_iface to z_int.

97fb80d

EAMxx: Using z_mam4_int.

6bc8e5c

EAMxx: Using z_mam4_int.

EAMxx: Using one year files for linoz and invariants.

a00ef52

EAMxx: Deleting code for former reader.

2fed8fd

EAMxx: Deleting code for former reader (Linoz and invariants).

e0609e9

EAMxx: Removing code from tracer_reader_utils.

1717a46

EAMxx: Removing unused code; including mam4_linoz_ymd, mam4_oxid_ymd,…

851c892

… and elevated_emiss_ymd from namelist.

EAMxx: Update linoz and oxid files in cmake and input.yaml.

8bf6a7b

EAMxx: Adding oxid_ to avoid conflicts with gases treated as tracers.

2998c04

EAMxx: Fixes after rebasing.

ea20f4d

odiazib force-pushed the eamxx/data_interpolation_microphysics branch from 05cfaf9 to ea20f4d Compare September 15, 2025 22:14

EAMxx: Add info to logger.

8b98bc4

EAMxx: Data interpolation for MAM4 microphysics. #7501

Are you sure you want to change the base?

EAMxx: Data interpolation for MAM4 microphysics. #7501

Uh oh!

Conversation

odiazib commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests (DataInterpolation for elevated emissions)

Tests (DataInterpolation for both invariants and linoz)

NC files

Elevated emissions

Invariants

Linoz

Uh oh!

bartgol left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

singhbalwinder left a comment

Choose a reason for hiding this comment

Uh oh!

singhbalwinder left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TaufiqHassan left a comment

Choose a reason for hiding this comment

Uh oh!

TaufiqHassan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

odiazib commented Jul 10, 2025 •

edited

Loading