Running GFDL microphysics with hydrostatic option

[benwgreen]

Description

I am trying to run SFS (in hydrostatic mode) with GFDL microphysics. This required me to create my own suite definition file that is modeled on suite_FV3_GFS_v17_coupled_p8_ugwpv1.xml, but modified for GFDL MP instead of Thompson. When I run the code, I get a segmentation fault error that nicely traces to the following:
281: 2 0x000000000292887f fv_sat_adj_mp_wqs2_vect_() /scratch2/BMC/gsd-fv3-dev/Ben.Green/SFS_GFDL_HY_20250407/sorc/ufs_model.fd/FV3/ccpp/physics/physics/MP/GFDL/fv_sat_adj.F90:1278
281: 3 0x0000000002924dd3 fv_sat_adj_mp_fv_sat_adj_work_() /scratch2/BMC/gsd-fv3-dev/Ben.Green/SFS_GFDL_HY_20250407/sorc/ufs_model.fd/FV3/ccpp/physics/physics/MP/GFDL/fv_sat_adj.F90:733
On Line 1278 (note: this is Line 1260 in the source code, I had added some print statements), I determined that a NaN was being fed into "it", which when fed into the lookup tables caused them to crash.

I do not get this error in non-hydrostatic mode. Moreover, I added print statements and found that there were NaNs and zeros in the variable "peln", thus "dpln" on Line 340 is NaN or zero, thus "den" on Line 511 has divide by NaN / divide by zero, etc.

In short, the problem is that the variable "peln" does not appear to be sent to fv_sat_adj.F90 correctly.

Steps to Reproduce

Please provide detailed steps for reproducing the issue.

1. Check out and compile the latest global-workflow (95be61086815f3a311b4385484b7f4436efa62e0) in hydrostatic mode. It is crucial that hydrostatic mode be used. It is not crucial to use the coupled model; it should happen for an atmosphere-only run with hydrostatic and GFDL MP.
2. Change the SDF to one that uses GFDL microphysics.
3. Run and see the crash...

Additional Context

Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

• Machine: Hera
• Compiler
• Suite Definition File or Scheme: Custom (I can provide upon requests)
• Reference other issues or PRs in other repositories that this is related to, and how they are related.

Output

In the log file:
281: 2 0x000000000292887f fv_sat_adj_mp_wqs2_vect_() /scratch2/BMC/gsd-fv3-dev/Ben.Green/SFS_GFDL_HY_20250407/sorc/ufs_model.fd/FV3/ccpp/physics/physics/MP/GFDL/fv_sat_adj.F90:1278
281: 3 0x0000000002924dd3 fv_sat_adj_mp_fv_sat_adj_work_() /scratch2/BMC/gsd-fv3-dev/Ben.Green/SFS_GFDL_HY_20250407/sorc/ufs_model.fd/FV3/ccpp/physics/physics/MP/GFDL/fv_sat_adj.F90:733

[benwgreen]

Update: In order to run hydrostatic FV3 with GFDL microphysics in UFS, it was noted that the flag "na_init" be set to 0. In making this change, my previous error no longer appears. So I will close this issue with the cautionary note about the need to set "na_init" to 0.

[benwgreen]

I've reopened this issue based on the feedback from ufs-weather-model PR#2503.

Both the original GFDLMP, and GFDLMPv3, will have the same problem in fv_sat_adj.F90 for na_init > 0 in hydrostatic mode, which is what @XiaqiongZhou-NOAA says that flag needs to be set to...

[XiaqiongZhou-NOAA]

Just did a quick debug and the model crashed with the following error message:
103: forrtl: severe (408): fort: (2): Subscript #1 of the array Q_CON has value 2 which is greater than the upper bound of 1

It makes sense now. For hydrostatic runs, DUSE_COND is off when compiling since this option may not work correctly for hydrostatic runs. fast_sat_adj is not dealing with q_con correctly when the model is hydro and not Use_cond.

[XiaqiongZhou-NOAA]

@benwgreen
Regarding the issue with running the hydrostatic configuration using GFDLMP,
I can confirm that it stemmed from mismatched dimensions of Q_con and delz between hydrostatic and non-hydrostatic runs. I was able to get the model running by modifying the dimensions on the dynamic core side. While the proper solution would be to address this on the CCPP side, I now have a working version as a temporary fix.

[benwgreen]

@XiaqiongZhou-NOAA
This does not make sense; perhaps there is some missing information?
Specifically, "delz" only appears in non-hydrostatic mode, specifically:

Lines 364, 367, and 369 are all under an if ( .not. hydrostatic ) then block on Line 360

Line 515 is under an else block for if (hydrostatic) then (cf. Lines 509 and 513)

Thus, why should delz matter when the hydrostatic flag is on?
As I said in the first post on this issue, the problem is traced back to the variable dpln (only used on the RHS on Line 511, and that is only when the hydrostatic flag is on).
dpln is always calculated from peln regardless of hydrostatic or non-hydrostatic (Line 340), but for the non-hydrostatic case the bad values of peln and dpln never get used.

[XiaqiongZhou-NOAA]

@benwgreen
There is mismatch when call fv_sat_adj_run in ccpp_FV3_GFS_v17_coupled_p8_ugwpv1_gfdlmp_fast_physics_cap.F90 since it defines delz size based on fv_sat_adj_run.meta.

[XiaqiongZhou-NOAA]

@benwgreen
My temporary fix is here. diff between the original and revised fv_arrays.F90 in dycore:
<
diff -r atmos_cubed_sphere/model/fv_arrays.F90 /scratch2/NCEPDEV/ensemble/Xiaqiong.Zhou/noscrub/gfs/gw-20250226/sorc/ufs_model.fd/FV3/atmos_cubed_sphere/model/fv_arrays.F90
1540,1541c1540
< allocate ( Atm%delz(is:ie, js:je ,1) )

   allocate ( Atm%delz(is:is, js:js  ,1) )

1558,1559c1557

< allocate ( Atm%q_con(isd:ied,jsd:jed,1) )

  allocate ( Atm%q_con(isd:isd,jsd:jsd,1) )

"