The coupled debug test is failing

[uturuncoglu]

The newly introduced coastal_ike_shinnecock_atm2sch2ww3_debug_intel test is failing with following error,

 8: [hercules-01-52:1829549:0:1829549] Caught signal 8 (Floating point exception: floating-point invalid operation)
 8: ==== backtrace (tid:1829549) ====
 8:  0 0x0000000000054d90 __GI___sigaction()  :0
 8:  1 0x0000000003baf986 schism_init_()  /work2/noaa/nems/tufuk/COASTAL/ufs-coastal/SCHISM-interface/SCHISM/src/Hydro/schism_init.F90:7028
 8:  2 0x000000000394c0a2 schism_nuopc_cap_mp_initializeadvertise_()  /work2/noaa/nems/tufuk/COASTAL/ufs-coastal/SCHISM-interface/SCHISM-ESMF/src/schism/schism_nuopc_cap.F90:345
 8:  3 0x0000000000b948f4 ESMCI::FTable::callVFuncPtr()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMCI_FTable.C:2167
 8:  4 0x0000000000b9886f ESMCI_FTableCallEntryPointVMHop()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMCI_FTable.C:824
 8:  5 0x0000000000d0f497 ESMCI::VMK::enter()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Infrastructure/VM/src/ESMCI_VMKernel.C:1125
 8:  6 0x0000000000cfb8f2 ESMCI::VM::enter()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Infrastructure/VM/src/ESMCI_VM.C:1216
 8:  7 0x0000000000b95d4a c_esmc_ftablecallentrypointvm_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMCI_FTable.C:981
 8:  8 0x00000000005b7b10 esmf_compmod_mp_esmf_compexecute_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMF_Comp.F90:1223
 8:  9 0x000000000089a7f1 esmf_gridcompmod_mp_esmf_gridcompinitialize_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMF_GridComp.F90:1412
 8: 10 0x0000000000b1cee0 nuopc_driver_mp_loopmodelcompss_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/addon/NUOPC/src/NUOPC_Driver.F90:2886
 8: 11 0x0000000000b45144 nuopc_driver_mp_initializeipdv02p1_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/addon/NUOPC/src/NUOPC_Driver.F90:1313
 8: 12 0x0000000000b4ef9b nuopc_driver_mp_initializegeneric_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/addon/NUOPC/src/NUOPC_Driver.F90:481
 8: 13 0x0000000000b948f4 ESMCI::FTable::callVFuncPtr()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMCI_FTable.C:2167
 8: 14 0x0000000000b9886f ESMCI_FTableCallEntryPointVMHop()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMCI_FTable.C:824
 8: 15 0x0000000000d0f28a ESMCI::VMK::enter()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Infrastructure/VM/src/ESMCI_VMKernel.C:2321
 8: 16 0x0000000000cfb8f2 ESMCI::VM::enter()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Infrastructure/VM/src/ESMCI_VM.C:1216
 8: 17 0x0000000000b95d4a c_esmc_ftablecallentrypointvm_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMCI_FTable.C:981
 8: 18 0x00000000005b7b10 esmf_compmod_mp_esmf_compexecute_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMF_Comp.F90:1223
 8: 19 0x000000000089a7f1 esmf_gridcompmod_mp_esmf_gridcompinitialize_()  /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.5.1/cache/build_stage/spack-stage-esmf-8.5.0-ffgsxo7vntsk3fr5lwbjnjrrviof6dz5/spack-src/src/Superstructure/Component/src/ESMF_GridComp.F90:1412
 8: 20 0x000000000042d13f MAIN__()  /work2/noaa/nems/tufuk/COASTAL/ufs-coastal/driver/UFS.F90:381
 8: 21 0x0000000000429f3d main()  ???:0
 8: 22 0x000000000003feb0 __libc_start_call_main()  ???:0
 8: 23 0x000000000003ff60 __libc_start_main_alias_2()  :0
 8: 24 0x0000000000429e55 _start()  ???:0
 8: =================================
 8: forrtl: error (75): floating point exception
 8: Image              PC                Routine            Line        Source             
 8: libc.so.6          0000147ABC625D90  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000003BAF986  schism_init_             7028  schism_init.F90
 8: fv3_coastal.exe    000000000394C0A2  schism_nuopc_cap_         345  schism_nuopc_cap.F90
 8: fv3_coastal.exe    0000000000B948F4  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B9886F  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000D0F497  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000CFB8F2  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B95D4A  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    00000000005B7B10  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    000000000089A7F1  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B1CEE0  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B45144  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B4EF9B  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B948F4  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B9886F  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000D0F28A  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000CFB8F2  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    0000000000B95D4A  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    00000000005B7B10  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    000000000089A7F1  Unknown               Unknown  Unknown
 8: fv3_coastal.exe    000000000042D13F  MAIN__                    381  UFS.F90
 8: fv3_coastal.exe    0000000000429F3D  Unknown               Unknown  Unknown
 8: libc.so.6          0000147ABC610EB0  Unknown               Unknown  Unknown
 8: libc.so.6          0000147ABC610F60  __libc_start_main     Unknown  Unknown
 8: fv3_coastal.exe    0000000000429E55  Unknown               Unknown  Unknown

[uturuncoglu]

@platipodium @josephzhang8 @pvelissariou1 @janahaddad I wonder if you are testing any coupled configuration build with standard debug options. The options used by ufs-weather-model can be seen in https://github.com/ufs-community/ufs-weather-model/blob/47c00995706380f9827a7e98bd242ffc6e4a7f01/cmake/Intel.cmake#L18. Anyway, please let me know if you have any idea. In the mean time, I'll test followings,

• test only atm-sch configuration to see what happens
• the failure is created by Intel compiler, it would be nice to test with GNU too
• it is also on Hercules, it would be nice to test another platform

[josephzhang8]

Add @danishyo