Reproducibility fails with 0.4.0 build and the latest configuration

[minghangli-uni]

Below are the 5-day ocean.stats results highlighting the repro issue (an issue is raised here https://github.com/COSIMA/access-om3/issues/265) when comparing the 0.3.0 and 0.4.0 builds with the 0.25 ryf configuration.

---

Using this existing configuration 5842715 and the 0.3.0 build, ocean.stats shows,

Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,     0, En 1.1422421593679974E-03, CFL  0.12730, SL  6.8277E-10, M 1.36283E+21, S 34.7206, T  3.6237, Me -3.81E-19, Se -2.83E-17, Te  8.84E-19
   127,  693137.000,     0, En 1.1320364471729537E-03, CFL  0.11451, SL  6.8348E-10, M 1.36283E+21, S 34.7206, T  3.6236, Me  1.59E-19, Se  1.36E-17, Te -1.75E-18
   191,  693138.000,     0, En 1.1510039944012112E-03, CFL  0.09915, SL  6.8065E-10, M 1.36283E+21, S 34.7206, T  3.6236, Me  7.92E-20, Se  5.60E-19, Te  2.54E-18
   255,  693139.000,     0, En 1.1290348792421638E-03, CFL  0.09146, SL  6.8277E-10, M 1.36283E+21, S 34.7206, T  3.6237, Me  4.93E-19, Se  1.61E-17, Te -2.30E-18
   319,  693140.000,     0, En 1.1255847638941558E-03, CFL  0.09269, SL  6.8348E-10, M 1.36283E+21, S 34.7206, T  3.6237, Me  7.10E-20, Se  1.03E-17, Te  1.02E-18

Using this updated configuration #169 with the 0.4.0 build, reproducibility fails despite having passed the existing repro test. For this specific case, 33 truncations occur on the 2nd day. But when using the latest topography and grid, the problem worsens, with over 20,000 truncation errors pop up ... (not shown here)

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,    33, En 1.1407486098043798E-03, CFL  0.12627, SL -3.4201E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me -9.42E-10, Se -4.89E-08, Te  5.68E-09
   127,  693137.000,     0, En 1.1294391817635600E-03, CFL  0.12267, SL -1.1492E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.25E-10, Se  1.30E-08, Te -3.45E-09
   191,  693138.000,     0, En 1.1501394646293486E-03, CFL  0.09017, SL  1.5319E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  7.38E-10, Se  2.36E-08, Te -2.95E-08
   255,  693139.000,     0, En 1.1299074397619121E-03, CFL  0.07864, SL  2.7025E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  3.22E-10, Se  1.11E-08, Te -4.45E-08
   319,  693140.000,     0, En 1.1271300577437651E-03, CFL  0.08655, SL  2.1515E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -1.52E-10, Se -4.01E-09, Te -1.84E-08

I was suspecting the issue might be related to the CICE grid, so I reverted to the old ice grid while keeping the 0.4.0 build. Then no truncations observed, but reproducibility failures persist...

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,     0, En 1.1422258589214400E-03, CFL  0.12731, SL -3.3210E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -9.15E-10, Se -4.80E-08, Te  5.75E-09
   127,  693137.000,     0, En 1.1319585613515698E-03, CFL  0.11500, SL -1.3026E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  5.56E-10, Se  1.08E-08, Te -4.95E-09
   191,  693138.000,     0, En 1.1508696289224552E-03, CFL  0.09993, SL  1.1858E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.85E-10, Se  2.18E-08, Te -3.17E-08
   255,  693139.000,     0, En 1.1288719261424492E-03, CFL  0.09160, SL  2.2845E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me  3.03E-10, Se  1.05E-08, Te -4.43E-08
   319,  693140.000,     0, En 1.1254764686950535E-03, CFL  0.09312, SL  1.9680E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -8.72E-11, Se -1.68E-09, Te -1.64E-08

So, it means the issue is likely not related to the cice grid alone, but also the latest 0.4.0 build? Or this is what 0.4.0 intentionally does? ping @anton-seaice @dougiesquire @chrisb13

[minghangli-uni]

You may find the 3rd line in each ocean.stats is always the same.

[anton-seaice]

Do you have MOM_parameter_doc.all from 0.3.0 and 0.4.0, are their differences ?

[minghangli-uni]

Good question. No, they are not the same for some parameters. Some parameters that were default false in 0.3.0 are now default true!

I’ll check them in detail, ensure they are the same and run another test to see if they are consistent.

[minghangli-uni]

Below these parameters have been modified to ensure consistency with MOM parameters, but the bfb repro still fails. Here's where the differences are: https://github.com/minghangli-uni/access-om3-doc/blob/compare_0.3.0_0.4.0/tables/MOM_parameter_all_0.3.0-0.4.0/MOM6-CICE6-0.3.0-0.4.0_nml_diff.md

USE_POROUS_BARRIER = True       !   [Boolean] default = False
                                ! If true, use porous barrier to constrain the widths and face areas at the 
                                ! edges of the grid cells.

RESOLN_N2_FILTER_DEPTH = 2000.0   !   [m] default = -1.0
                                ! The depth below which N2 is monotonized to avoid stratification artifacts from
                                ! altering the equivalent barotropic mode structure.  This monotonzization is
                                ! disabled if this parameter is negative.
REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! This selects the remapping algorithm used in OM4 that does not use the full
                                ! reconstruction for the top- and lower-most sub-layers, but instead assumes
                                ! they are always vanished (untrue) and so just uses their edge values. We
                                ! recommend setting this option to false.

DIAG_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for diagnostics. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting this option to
                                ! false.

EBT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

HBD_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for horizontal boundary
                                ! diffusion. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

INTWAVE_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

Z_INIT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for initialization. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for more details. We recommend setting this option
                                ! to false.

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err 
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 6.3048230599477494E-22, CFL  0.00000, SL  6.8348E-10, M 1.36283E+21, S 34.7205, T  3.6237, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    63,  693136.000,     0, En 1.1424681980749408E-03, CFL  0.12731, SL -3.3095E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -9.11E-10, Se -4.84E-08, Te  1.32E-09
   127,  693137.000,     0, En 1.1322062972728886E-03, CFL  0.11446, SL -1.0190E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.31E-10, Se  1.28E-08, Te -6.21E-09
   191,  693138.000,     0, En 1.1511417771868093E-03, CFL  0.09976, SL  1.3491E-06, M 1.36283E+21, S 34.7206, T  3.6236, Me  6.52E-10, Se  2.07E-08, Te -3.15E-08
   255,  693139.000,     0, En 1.1292283529569494E-03, CFL  0.09152, SL  2.4163E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me  2.94E-10, Se  1.00E-08, Te -4.56E-08
   319,  693140.000,     0, En 1.1258543366824954E-03, CFL  0.09256, SL  1.7924E-06, M 1.36283E+21, S 34.7206, T  3.6237, Me -1.72E-10, Se -4.66E-09, Te -1.75E-08

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/cosima-twg-announce/401/56

[aekiss]

No, they are not the same for some parameters. Some parameters that were default false in 0.3.0 are now default true!

If we stored up-to-date MOM_parameter_doc.* in the repo we'd catch these problems https://github.com/COSIMA/access-om3/issues/117

[dougiesquire]

Below these parameters have been modified to ensure consistency with MOM parameters, but the bfb repro still fails. Here's where the differences are: https://github.com/minghangli-uni/access-om3-doc/blob/compare_0.3.0_0.4.0/tables/MOM_parameter_all_0.3.0-0.4.0/MOM6-CICE6-0.3.0-0.4.0_nml_diff.md

USE_POROUS_BARRIER = True       !   [Boolean] default = False
                                ! If true, use porous barrier to constrain the widths and face areas at the 
                                ! edges of the grid cells.

RESOLN_N2_FILTER_DEPTH = 2000.0   !   [m] default = -1.0
                                ! The depth below which N2 is monotonized to avoid stratification artifacts from
                                ! altering the equivalent barotropic mode structure.  This monotonzization is
                                ! disabled if this parameter is negative.
REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! This selects the remapping algorithm used in OM4 that does not use the full
                                ! reconstruction for the top- and lower-most sub-layers, but instead assumes
                                ! they are always vanished (untrue) and so just uses their edge values. We
                                ! recommend setting this option to false.

DIAG_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for diagnostics. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting this option to
                                ! false.

EBT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

HBD_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for horizontal boundary
                                ! diffusion. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

INTWAVE_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for calculating EBT 
                                ! structure. See REMAPPING_USE_OM4_SUBCELLS for details. We recommend setting
                                ! this option to false.

Z_INIT_REMAPPING_USE_OM4_SUBCELLS = False !   [Boolean] default = True
                                ! If true, use the OM4 remapping-via-subcells algorithm for initialization. See 
                                ! REMAPPING_USE_OM4_SUBCELLS for more details. We recommend setting this option
                                ! to false.

Noting for documentation purposes that this list is not correct.

• The *_OM4_SUBCELLS parameters should be set to True to retain the original behaviour.
• Changing USE_POROUS_BARRIER to True does nothing in our case (other than to slow the model down) since we don't configure any porous barriers. See Change default of USE_POROUS_BARRIER to false NOAA-GFDL/MOM6#735
• Changing RESOLN_N2_FILTER_DEPTH to the previous default of 2000.0 actually changes answers. There was previously a bug that meant that RESOLN_N2_FILTER_DEPTH was actually being discarded. The new default disables monotonization, consistent with the previous buggy behaviour. See +(*)Fix wave_speed_init mono_N2_depth bug NOAA-GFDL/MOM6#413

There is actually only one change to the MOM6 parameters needed to reproduce MOM6 answers (see https://github.com/COSIMA/access-om3/issues/270#issuecomment-2658161227):

USE_HUYNH_STENCIL_BUG = True    !   [Boolean] default = False
                                ! If true, use a stencil width of 2 in PPM:H3 tracer advection. This is
                                ! incorrect and will produce regressions in certain configurations, but may be
                                ! required to reproduce results in legacy simulations.

However, even with this parameter set, ACCESS-OM3 is not reproducible across 0.3.1 and 0.4.0 due to reproducibility-breaking changes in CICE, CMEPS and CDEPS - see #173

[anton-seaice]

@minghangli-uni - did you look at reproducibility over restarts at all? I tried running the model-config-tests restart reproducability and it also failed:

[as2285@gadi-login-03 archive]$ cat MOM6-CICE6-B-test_restart_repro_2day/output000/ocean.stats
  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 3.1034198253312566E-23, CFL  0.00000, SL  1.5183E-10, M 1.36404E+21, S 34.7263, T  3.6362, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    23,  693136.000,     0, En 1.6467884517688178E-03, CFL  0.08144, SL -1.3225E-05, M 1.36404E+21, S 34.7263, T  3.6363, Me -3.63E-09, Se -1.67E-07, Te -3.44E-06
    47,  693137.000,     0, En 1.7271582761777793E-03, CFL  0.06106, SL -5.1177E-06, M 1.36404E+21, S 34.7263, T  3.6364, Me  2.22E-09, Se  1.40E-07, Te -2.71E-06
[as2285@gadi-login-03 archive]$ cat MOM6-CICE6-B-test_restart_repro_2x1day/output000/ocean.stats
  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err
            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]
     0,  693135.000,     0, En 3.1034198253312566E-23, CFL  0.00000, SL  1.5183E-10, M 1.36404E+21, S 34.7263, T  3.6362, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    23,  693136.000,     0, En 1.6467884517688178E-03, CFL  0.08144, SL -1.3225E-05, M 1.36404E+21, S 34.7263, T  3.6363, Me -3.63E-09, Se -1.67E-07, Te -3.44E-06
[as2285@gadi-login-03 archive]$ cat MOM6-CICE6-B-test_restart_repro_2x1day/output001/ocean.stats
     0,  693136.000,     0, En 1.6468021784271944E-03, CFL  0.08144, SL -1.3225E-05, M 1.36404E+21, S 34.7263, T  3.6363, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00
    24,  693137.000,     0, En 1.7269455930905098E-03, CFL  0.06104, SL  1.5465E-05, M 1.36404E+21, S 34.7263, T  3.6364, Me  7.87E-09, Se  4.39E-07, Te -4.13E-07

Key points:

Did 2x1 day runs and 1 2 day run.

Results at the end of the first day seem the same

Some fields at the end of the second day are different. The error fields go back to 0 at the start of the second day in the 2x1 day run ? Is that expected ?

Still appears to not be very good at counting 24 hours !

[minghangli-uni]

Are these from 0.3.0 or 0.4.0? Restart repro works in 0.3.0 right? I haven’t checked 0.4.0 yet.

The error fields go back to 0 at the start of the second day in the 2x1 day run ? Is that expected ?

Yes, the error terms (Frac Mass Err, Salin Err, Temp Err) are recalculated independently for each run. They dont persist across runs cuz they are used to track errors within a single run but not across multiple runs. We should focus on terms beforehand.

There should be a separate restart repro check for each component. I dont think it comes from mom6 but might come from other components.

[dougiesquire]

@minghangli-uni - did you look at reproducibility over restarts at all? I tried running the model-config-tests restart reproducability and it also failed

Rats. I spy another rabbit hole

[anton-seaice]

I confirmed the result using the CI:

see #177

The failure is comparing 2x1 day consecutive runs to 1x2 day run

There are three issues at a first glance:

• comparing error fields in ocean.stats will always give different answers as they change with different start points in time (Frac Mass Err, Salin Err, Temp Err)
• The two day run produces a ocean.stats with 3 lines in it, and the 1 day runs produces 2 ocean.stats with 2 lines in them, and we are trying to compare the all the lines. We should be trying to compare the last lines? (or creating the correct three lines from the 2x1 day runs).
• En, CFL and SL are all different after two days.