New spatially-variable horizontal upwind capability #7371

cbegeman · 2025-05-20T20:26:03Z

Here we add a new capability to transition between centered and upwinded horizontal thickness advection as a function of column thickness.

Formerly, either thickness advection was centered or upwinded everywhere. With this PR, when config_use_spatially_variable_upwinding is .true., advection is fully upwinded when the column thickness is less than config_spatially_variable_upwind_hmin and fully centered when column thickness is greater than config_spatially_variable_upwind_hmax. Between these values, there is a linear transition layerThickEdgeFlux

[NML]
[BFB] stealth feature

cbegeman · 2025-05-20T20:28:06Z

Testing

Parabolic bowl test has been run and found to have similar convergence and error.

Drying slope ramp test also qualitatively similar when ramp extends from thin film thickness to 1m and upwinded to centered transitions between water column thickness of 1m - 5m.

IcoswISC240/WOA23/performance_test runs with diffs when spatially variable upwinding is on between 5m - 50m

Passes the e3sm_cryo_developer suite on Chrysalis with the feature off

Passes the SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5.chrysalis_intel.mpaso-upwind_advection with the feature on, ramping between 10 and 50m, and the time integration scheme changed to split_explicit. Fails with split_explicit_ab2 with CFL violation in MPAS-Seaice indicating an instability.

cbegeman · 2025-05-20T20:29:36Z

@sbrus89 Would you mind giving the code a look and letting me know whether you would be satisfied for this to go in without the results of your hurricane test?

cbegeman · 2025-06-03T19:05:40Z

@sbrus89 Let me know if you have any questions about this PR

cbegeman · 2025-06-17T16:45:45Z

@xylar and @sbrus89 please review when you have a chance. Thanks!

sbrus89

@cbegeman, This looks good by inspection. I just have one minor comment. Sorry for the delay on the review.

sbrus89 · 2025-06-17T16:59:07Z

components/mpas-ocean/src/shared/mpas_ocn_diagnostics.F

@@ -985,7 +985,43 @@ subroutine ocn_diagnostic_solve_layerThicknessEdge(normalVelocity, &
 #endif

        end if
-
+        if (config_use_spatially_variable_upwind) then


There should probably be a check to make sure in the init routine to require config_thickness_flux_type = 'upwind' with config_use_spatially_variable_upwind=.true.. I suppose there's no harm in leaving config_use_spatially_variable_upwind=.true. on for config_thickness_flux_type = 'centered', but right now it will silently do a needless interpolation between the same values.

Good point. I'll add that.

@sbrus89 This check is in case (thickEdgeFluxUpwind) so I don't think there is a needless interpolation. Let me know if you see something that I don't.

Ah, you're right; I missed that. Thanks!

components/mpas-ocean/src/shared/mpas_ocn_diagnostics.F

components/mpas-ocean/src/Registry.xml

xylar

I think this looks good so far! It needs a stealth test. @cbegeman, would you like help adding one?

components/mpas-ocean/src/Registry.xml

xylar · 2025-06-18T13:19:11Z

An obvious example would be:
https://github.com/E3SM-Project/E3SM/blob/master/cime_config/tests.py#L301
https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/cime_config/testdefs/testmods_dirs/mpaso/upwind_advection/user_nl_mpaso

xylar · 2025-06-18T14:30:17Z

I'm going to test this out with a short (5 day) SORRMr4 run where I'm writing out upwindFactor in the history files. Let me know if that sounds reasonable.

cbegeman · 2025-06-18T14:32:22Z

I'm going to test this out with a short (5 day) SORRMr4 run where I'm writing out upwindFactor in the history files. Let me know if that sounds reasonable.

@xylar Go for it. If you want upwinding to be on, just make sure that the upwinding range isn't below the minimum wct for that mesh.

…g_slope results

xylar · 2025-06-18T15:05:41Z

If you want upwinding to be on, just make sure that the upwinding range isn't below the minimum wct for that mesh.

Very good point!

xylar · 2025-06-18T15:31:39Z

@cbegeman, I tried to run with:

config_thickness_flux_type = 'upwind'
config_use_spatially_variable_upwind = .true.
config_spatially_variable_upwind_hmin = 10.0
config_spatially_variable_upwind_hmax = 20.0

The run is here:

/lcrc/group/e3sm/ac.xylar/scratch/chrys/20250618.GMPAS-JRA1p5-DIB-PISMF.TL319_SOwISC12to30E3r4.upwind_factor.chrysalis/run

It crashed about 10 minutes in with a state validation error.

I can debug on Friday if needed but probably won't get to it before then. Just wanted to point it out in case you want to look into it earlier.

I'll run the same test without upwinding to make completely sure it works.
Update: No, I was able to run fine without upwinding, see:

/lcrc/group/e3sm/ac.xylar/scratch/chrys/20250618.GMPAS-JRA1p5-DIB-PISMF.TL319_SOwISC12to30E3r4.chrysalis/run

cbegeman · 2025-06-18T16:42:58Z

@xylar I'll see if I can run compass's ocean/global_ocean/SOwISC12to30/MALI_topo_AIS_4to20km. I am not sure whether a new dynamic adjustment would be needed.

xylar · 2025-06-18T16:44:54Z

I am not sure whether a new dynamic adjustment would be needed.

Ah, that certainly does seem plausible. If so, just running with super short time steps is worth a go.

Update: Reducing the time step even to 10 seconds from 10 minutes didn't help. I'm seeing a crash around 0001-01-01_01:10:00 in every simulation. (So about 40 minutes in, as the simulation appears to start at 30 minutes into day 1.) I misspoke above.

I'm also seeing annoying hangs without the job ending as reported in:
#7447

xylar · 2025-06-27T14:56:03Z

@cbegeman, I tried running:

./create_test SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5.chrysalis_intel.mpaso-upwind_advection --wait

I'm seeing errors like:

 80: forrtl: error (72): floating overflow
 80: Image              PC                Routine            Line        Source  
 80: libpnetcdf.so.3.0  000015554B27962C  for__signal_handl     Unknown  Unknown
 80: libpthread-2.28.s  00001555473D0CF0  Unknown               Unknown  Unknown
 80: e3sm.exe           0000000005CC1D32  Unknown               Unknown  Unknown
 80: e3sm.exe           0000000003143250  ocn_diagnostics_m        2967  mpas_ocn_diagnostics.f90

This is pointing at the line:

         fracAbsorbed = 1.0_RKIND &
                      - exp( max(-100.0_RKIND, &
                                 -layerThickness(kmin, iCell)/ &
                           config_flux_attenuation_coefficient))

I verified that the test runs fine when I remove .mpaso-upwind_advection. I tried reducing the hmax:

 config_spatially_variable_upwind_hmax = 30.0

But that didn't change the results (perhaps unsurprisingly).

I suspect we might be seeing negative layer thicknesses or something like that, but I haven't yet dived deeper.

Let me know if you have thoughts on this.

cbegeman · 2025-07-01T18:07:00Z

@xylar Using the split_explicit time integrator, the IcoswISC mesh passes (1 month simulation) while the SORRM mesh still fails.

Instability with the SORRM mesh likely does not indicate an issue with this feature but rather that we also need wetting-and-drying to be active. As such, I think we can proceed with making this a stealth feature.

xylar

@cbegeman, sounds good to me.

I reran:

./create_test SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5.chrysalis_intel.mpaso-upwind_advection --wait

and with the switch to split-explicit time stepping, it now works for me, too.

I agree that we can investigate SORRM further in combination with wetting-and-drying updates that you have waiting in the wings.

sbrus89

Approved based on inspection and others' testing.

cbegeman · 2025-07-02T14:32:09Z

@xylar and @sbrus89 Thank you both for reviewing!

@jonbob This is ready for your testing.

jonbob · 2025-07-02T16:02:10Z

@cbegeman -- if this is a stealth feature, we need to add a test to the e3sm_ocnice_stealth_features suite? Or does this reuse the SMS_D_Ld1.T62_oQU240.GMPAS-IAF.mpaso-upwind_advection test that's already there?

jonbob · 2025-07-02T16:07:33Z

@cbegeman -- does this PR need the scripts run to make the bld files consistent with Registry? I don't see changes to build-namelist

cbegeman · 2025-07-02T17:12:09Z

@cbegeman -- if this is a stealth feature, we need to add a test to the e3sm_ocnice_stealth_features suite? Or does this reuse the SMS_D_Ld1.T62_oQU240.GMPAS-IAF.mpaso-upwind_advection test that's already there?

It does reuse that case. I think it fulfills testing upwind advection to apply upwinding to part of the domain with spatially vairable upwinding.

cbegeman · 2025-07-02T17:14:43Z

@cbegeman -- does this PR need the scripts run to make the bld files consistent with Registry? I don't see changes to build-namelist

@jonbob I changed build-namelist-section. Is there a need to change build-namelist? Maybe it's worth running your script in case I missed something. Thanks!

jonbob · 2025-07-02T17:43:25Z

@cbegeman -- yes, it's necessary to make changes to build-namelist. The build-namelist-section file doesn't actually get used for anything but is mostly a product of the generate_e3sm_namelist_files script. It has to get mirrored in build-namelist itself to actually be used.

cbegeman · 2025-07-02T21:18:17Z

@jonbob Thanks! I seem to frequently forget at least one of the many places to update the namelist!

#7371) New spatially-variable horizontal upwind capability Here we add a new capability to transition between centered and upwinded horizontal thickness advection as a function of column thickness. Formerly, either thickness advection was centered or upwinded everywhere. With this PR, when config_use_spatially_variable_upwinding is .true., advection is fully upwinded when the column thickness is less than config_spatially_variable_upwind_hmin and fully centered when column thickness is greater than config_spatially_variable_upwind_hmax. Between these values, there is a linear transition layerThickEdgeFlux [NML] [BFB] stealth feature

jonbob · 2025-07-09T21:56:33Z

Passes:

SMS_D_Ld1.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.chrysalis_intel.allactive-wcprod
e3sm_cryo_developer

with expected NML DIFFs. Also successfully ran:

SMS_D_Ld1.TL319_IcoswISC30E3r5.GMPAS-JRA1p5.chrysalis_intel.mpaso-upwind_advection

merged to next

jonbob · 2025-07-10T15:43:58Z

merged to master and expected NML DIFFs blessed on all machine except pm-cpu, which has yet to report

jonbob · 2025-07-15T15:23:54Z

also blessed on pm-cpu

cbegeman · 2025-07-15T16:20:54Z

Thank you for all your testing and moving this forward, @jonbob!

cbegeman requested review from sbrus89 and xylar May 20, 2025 20:26

cbegeman assigned jonbob May 20, 2025

cbegeman added the Stealth PR has feature which, if turned on, could change climate. fka FCC label May 20, 2025

cbegeman added mpas-ocean BFB PR leaves answers BFB labels May 20, 2025

sbrus89 reviewed Jun 17, 2025

View reviewed changes

components/mpas-ocean/src/shared/mpas_ocn_diagnostics.F Show resolved Hide resolved

sbrus89 reviewed Jun 17, 2025

View reviewed changes

components/mpas-ocean/src/Registry.xml Show resolved Hide resolved

xylar reviewed Jun 18, 2025

View reviewed changes

components/mpas-ocean/src/Registry.xml Show resolved Hide resolved

cbegeman added 5 commits June 18, 2025 09:52

New spatially variable upwinding capability

89e7410

Set upwind factor in diagnostics every time step

2df3a12

Change minimum column thickness for centered-weighting based on dryin…

6c67371

…g_slope results

Update build files

eae3237

Add spatially variable upwinding to existing upwind test

d692dad

cbegeman force-pushed the ocn/use-horiz-field-for-upwinding branch from de06dc1 to d692dad Compare June 18, 2025 15:57

xylar mentioned this pull request Jun 23, 2025

Crashes due to MPAS-Ocean state validation do not stop E3SM or standalone MPAS-Ocean runs #7447

Closed

Change time integrator for upwind test to SE

9778206

cbegeman marked this pull request as ready for review July 1, 2025 18:07

xylar approved these changes Jul 2, 2025

View reviewed changes

sbrus89 approved these changes Jul 2, 2025

View reviewed changes

Make bld files consistent with Registry

2fec412

jonbob added the NML label Jul 9, 2025

jonbob merged commit dfb364d into E3SM-Project:master Jul 10, 2025
3 checks passed

cbegeman deleted the ocn/use-horiz-field-for-upwinding branch July 15, 2025 16:20

New spatially-variable horizontal upwind capability #7371

New spatially-variable horizontal upwind capability #7371

Uh oh!

Conversation

cbegeman commented May 20, 2025 • edited by jonbob Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

cbegeman commented May 20, 2025

Uh oh!

cbegeman commented Jun 3, 2025

Uh oh!

cbegeman commented Jun 17, 2025

Uh oh!

sbrus89 left a comment

Choose a reason for hiding this comment

Uh oh!

sbrus89 Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbegeman Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

cbegeman Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

sbrus89 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xylar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xylar commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Jun 18, 2025

Uh oh!

cbegeman commented Jun 18, 2025

Uh oh!

xylar commented Jun 18, 2025

Uh oh!

xylar commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented Jun 18, 2025

Uh oh!

xylar commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Jun 27, 2025

Uh oh!

cbegeman commented Jul 1, 2025

Uh oh!

xylar left a comment

Choose a reason for hiding this comment

Uh oh!

sbrus89 left a comment

Choose a reason for hiding this comment

Uh oh!

cbegeman commented Jul 2, 2025

Uh oh!

jonbob commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonbob commented Jul 2, 2025

Uh oh!

cbegeman commented Jul 2, 2025

Uh oh!

cbegeman commented Jul 2, 2025

Uh oh!

jonbob commented Jul 2, 2025

Uh oh!

cbegeman commented Jul 2, 2025

cbegeman commented May 20, 2025 •

edited by jonbob

Loading

cbegeman commented May 20, 2025 •

edited

Loading

sbrus89 Jun 17, 2025 •

edited

Loading

xylar commented Jun 18, 2025 •

edited

Loading

xylar commented Jun 18, 2025 •

edited

Loading

xylar commented Jun 18, 2025 •

edited

Loading

jonbob commented Jul 2, 2025 •

edited

Loading