Artificial domain decomposition imprint in initial time steps

[uturuncoglu]

I am seeing very strange behavior in DATM+SCHISM+WW3 configuration when I check @yunfangsun Atlantic case (coastal_ian_crs_atm2sch2ww3) with new co-processing component in a higher temporal resolution (creating plot in every coupling time step - I change it to every 10 min or 360 s).

The following plot belongs to sea surface height from SCHISM in the first coupling time step,

and following animated gif for first couple of time step. As you can see the issue is appearing in the initial couple of steps and then disappearing. Maybe, it is just an initial condition issue and maybe the setup is starting from rest (cold start). It is hard to tell without detail of the configuration.

@yunfangsun could you check your configuration and add any detail that might help. I also noticed that the sea surface temperature provided by SCHISM is also almost constant for 3-hours. I am not sure but there might be some issue with this configuration. If you could also create a SST time series from your run for a specific location, it would be great.

JFYI, @yunfangsun @mansurjisan @platipodium @josephzhang8

[platipodium]

One idea: since the partitioning shows up as values nearer to zero, it seems that we communicate too much or too many zeros as ghost elements from the model to cap to coprocessor. So we should in addition check that we prevent communicating accidental zeros in ghost cells ...

[xuchengov]

To do:
@mansurjisan will first run the case to re-produce the error and then try to run with different CPU allocations to see if the pattern changes, which can narrow down the scope of the issue.

[uturuncoglu]

@platipodium @josephzhang8 Okay. I was looking to this issue and I run the Duck case with NUOPC connectors and node based decomposition. The decomposition boundaries are seen also in that case. Then, I check the code that performs halo update using ESMF call and I realized that isDataPtr%haloHandle is not created anywhere in the code and halo update was never called. It seems it was not exist also in the old version of the cap (before I started to modify it). Please correct me if I am wrong. So, I am not sure what is the background to introduce that variable but it seems that it was not used before too. Anyway, I go ahead and create the route handle for the halo update. I will test with the same configuration to see if it solves that issue with the domain decomposition boundaries at least for the node based run. But, if anyone could give some information to me about the isDataPtr%haloHandle and it is usage in the past that would be really helpful since I am having trouble to understand why it was there and why the route handle was not created in the first place.

[platipodium]

Hi @uturuncoglu, as far as I recall, I introduced the haloHandle but did not complete its implementation because we thought at that time that we would not need to update the halo region. I am still not sure whether we need it at all (and I lean towards no), but please go ahead trying whether this fixes the case.

[josephzhang8]

Yes there was some confusion on our side after talking with Bob O. His doc seems to suggest we do not need to add halo on ESMF side. Thx @uturuncoglu !

[uturuncoglu]

@platipodium @josephzhang8 Thanks for the clarification. It is good to know that implementation is not complete. BTW, These kind of code in the model and cap make debugging really hard since I was trusting halo update is working. I think it would be nice to clean the cap and remove those extra codes if the implementation is not fully functional etc. The development work could stay in a development branch for clarity. Anyway, my intention is not to restructure the cap or fix all the issues but at least I'll try to fix this halo update.