EAMxx: dycore energy fixer #7553

oksanaguba · 2025-07-28T02:04:22Z

Adds (dycore) energy fixer. The infrastructure can be used for another process (and will be used for IEFLX),
but the current impl is for the dycore.
It is identical to the EAM impl, where pressure adjustment and dycore energy leaks are fixed as one
global change to temperature for each column/level as the same temperature tendency.

There is a check if fixer debug output is on ( enable_energy_fixer_debug_info ) ,
which computes global energy after the fixer to confirm
that it does match. This debug output is used in a test that has postproc *py script to
confirm that the energy is fixed up to a tolerance.

For full discussion see #7276 .

Climo results ( model vs model, with fixer and without from the same branch) https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.onguba/theta/fixerJuly24-2025/latlon-pdf-vs-default/viewer/lat_lon/index.html .

[nonBFB] for any cime runs with eamxx and dycore. Potentially climate-changing, too.

oksanaguba · 2025-07-28T20:50:32Z

Discussing with @bartgol results from CI tests:
gcc-openmp/* and gcc-cuda/* (4 suites total) all fail for baseline comp tests with homme with cprnc output in the form

  A total number of    245 fields were compared
          of which      0 had non-zero differences
               and     14 had differences in fill patterns
               and      0 had different dimension sizes
               and      0 had different data types

That is, fill patterns changed (dp, water species, z-based fields), because of code

  get_field_out("pseudo_density",pgn).get_header().get_tracking().update_time_stamp(start_of_step_ts());

that we needed initially to make cime tests with certain IO pass (otherwise dp is not init-ed properly before homme, but it is needed for the fixer).

Standalone sp and fpe tests do not run homme, this is why they pass.

Cime tests with baselines fail in comparison, but pass if there is no baseline comparison.
IMO CI testing looks ok then.

bartgol

I have some last minute comments.

bartgol · 2025-07-29T19:56:42Z

components/eamxx/src/share/atm_process/atmosphere_process_group.cpp

    }

+//OG do we really need this check? It gets triggered for homme, but i do not see much use of this, disabling.
+#if 0    


I would remove the chunk of code then, rather than ifdeffing it out...

done, thanks

bartgol · 2025-07-29T20:00:50Z

components/eamxx/cime_config/namelist_defaults_eamxx.xml

      <!-- Frequency in physics steps to output a global hash over the dycore's
           in-fields. <= 0 disables hashing. -->
+      <enable_column_conservation_checks>false</enable_column_conservation_checks>
+      <enable_energy_fixer>true</enable_energy_fixer>


Would you mind adding some metadata to these new XML entries? Namely, the "type" and "doc" metadata. We use these metadata entries when we auto-generate documentation for model inputs (which gets deployed on the online docs).

done, thanks

share/util/shr_reprosum_mod.F90

mahf708 · 2025-07-29T20:07:39Z

components/eamxx/cime_config/namelist_defaults_eamxx.xml

      <!-- Frequency in physics steps to output a global hash over the dycore's
           in-fields. <= 0 disables hashing. -->
+      <enable_column_conservation_checks>false</enable_column_conservation_checks>
+      <enable_energy_fixer>true</enable_energy_fixer>


do you have a preference for this being true as opposed to false for now?

The fixer should be on always, otherwise we have a model that leaks energy (though it is a question how much is leaked and at what resolutions, we only know a little about these numbers from ne30 EAM runs). I believe the idea was to have the fixer on for all runs.

Yeah, I agree, I meant from the perspective of integration. If you wanted to integrate all-BFB then in a follow-up PR do the NBFB, or just go at once. Either way is fine, up to you

tcclevenger

One small comment, feel free to ignore if you like.

components/eamxx/src/share/property_checks/mass_and_energy_conservation_check.hpp

jgfouca

I took a pretty close look at the new python. It looks good for the most part, just a cpl suggestions.

components/eamxx/scripts/check-fixer-output

Adds (dycore) energy fixer. The infrastructure can be used for another process (and will be used for IEFLX), but the current impl is for the dycore. It is identical to the EAM impl, where pressure adjustment and dycore energy leaks are fixed as one global change to temperature for each column/level as the same temperature tendency. There is a check if fixer debug output is on ( enable_energy_fixer_debug_info ) , which computes global energy after the fixer to confirm that it does match. This debug output is used in a test that has postproc *py script to confirm that the energy is fixed up to a tolerance. For full discussion see #7276 . Climo results ( model vs model, with fixer and without from the same branch) https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.onguba/theta/ fixerJuly24-2025/latlon-pdf-vs-default/viewer/lat_lon/index.html . [nonBFB] for any cime runs with eamxx and dycore. Potentially climate-changing, too.

oksanaguba · 2025-07-31T03:05:00Z

in next. i reintroduced #include "ekat/kokkos/ekat_kokkos_utils.hpp" in mass_and_energy...hpp in a conflict resolution. if this gets thru i will check if it is really needed and put other small fixes in a small PR as i promised a few people.

Adds (dycore) energy fixer. The infrastructure can be used for another process (and will be used for IEFLX), but the current impl is for the dycore. It is identical to the EAM impl, where pressure adjustment and dycore energy leaks are fixed as one global change to temperature for each column/level as the same temperature tendency. There is a check if fixer debug output is on ( enable_energy_fixer_debug_info ) , which computes global energy after the fixer to confirm that it does match. This debug output is used in a test that has postproc *py script to confirm that the energy is fixed up to a tolerance. For full discussion see #7276 . Climo results ( model vs model, with fixer and without from the same branch) https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.onguba/theta/ fixerJuly24-2025/latlon-pdf-vs-default/viewer/lat_lon/index.html . [nonBFB] for any cime runs with eamxx and dycore. Potentially climate-changing, too.

jgfouca

Oksana will do the python stuff in a subsequent PR.

oksanaguba added 5 commits July 27, 2025 20:34

rename conservation files

f16c96e

implement fixer

0f71895

nl changes for fixer

5c1db62

add fixer test and post proc script

30b27fc

delete renamed files

ed6043d

oksanaguba requested review from bartgol and mahf708 July 28, 2025 22:38

oksanaguba self-assigned this Jul 28, 2025

oksanaguba requested a review from tcclevenger July 28, 2025 22:38

oksanaguba added Atmosphere non-BFB PR makes roundoff changes to answers. CC PR is climate changing HOMME labels Jul 28, 2025

remove comments that are not relevant

5b382d1

oksanaguba changed the title ~~[eamxx] dycore energy fixer~~ EAMxx: dycore energy fixer Jul 29, 2025

bartgol reviewed Jul 29, 2025

View reviewed changes

mahf708 requested a review from jgfouca July 29, 2025 20:07

mahf708 reviewed Jul 29, 2025

View reviewed changes

oksanaguba added 2 commits July 29, 2025 15:25

add metadata for fixer entries

eb5a9b5

remove permamently disabled ifdef

0cbd887

mahf708 approved these changes Jul 29, 2025

View reviewed changes

tcclevenger approved these changes Jul 29, 2025

View reviewed changes

components/eamxx/src/share/property_checks/mass_and_energy_conservation_check.hpp Show resolved Hide resolved

bartgol approved these changes Jul 29, 2025

View reviewed changes

jgfouca requested changes Jul 30, 2025

View reviewed changes

components/eamxx/scripts/check-fixer-output Show resolved Hide resolved

components/eamxx/scripts/check-fixer-output Show resolved Hide resolved

components/eamxx/scripts/check-fixer-output Show resolved Hide resolved

fixes for new ekat

9c77714

jgfouca self-requested a review August 4, 2025 17:33

oksanaguba requested review from jgfouca and removed request for jgfouca August 4, 2025 18:32

jgfouca approved these changes Aug 4, 2025

View reviewed changes

oksanaguba merged commit d545d88 into master Aug 4, 2025
5 of 19 checks passed

oksanaguba deleted the oksanaguba/eamxx/fixer-clean branch August 4, 2025 18:38

EAMxx: dycore energy fixer #7553

EAMxx: dycore energy fixer #7553

Uh oh!

Conversation

oksanaguba commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oksanaguba commented Jul 28, 2025

Uh oh!

bartgol left a comment

Choose a reason for hiding this comment

Uh oh!

bartgol Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

oksanaguba Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

bartgol Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

oksanaguba Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mahf708 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

oksanaguba Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

mahf708 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

tcclevenger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jgfouca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oksanaguba commented Jul 31, 2025

Uh oh!

jgfouca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

oksanaguba commented Jul 28, 2025 •

edited

Loading