Implement Sectoral Water Usage in CTSM

[TaranuDev]

Expected behavior:

1. Water withdrawal and consumption is read from input files at monthly frequency. Sub-monthly withdrawal/consumption is assumed uniform, meaning that we divide monthly amount by number of days in that month.
2. The sectors implemented: domestic, manufacturing, thermoelectric, livestock and mining.
3. The application times for domestic and livestock are between local 6AM till midnight 12AM (total time of flux application is 18 hours).
4. The application times for industrial (thermoelectric, manufacturing and mining) is constant throughout the entire day (24 hours).
5. In terms of satisfying the withdrawal, we use as much as possible from river (default minimum river volume limit), then as much as possible from unconfined groundwater, and then two options, or take from ocean or reduced withdrawal (save unsatisfied withdrawal as output variable).
6. The return flow, calculated as the difference between withdrawal and consumption, returns to the river on the next time step.
7. The consumed water is added to surface soil only over natural vegetation and previous streets for urban areas. This is done to avoid interference with the irrigation module.

An initial coding plan is attached below.

CodePlan_v1.md

[TaranuDev]

Hello!

I would like to ask for some advice.
At the moment we did some nice progress on this project.
The sectoral withdrawal, consumption and return flow fluxes are computed daily inside CTSM taking into account water available in the river and also featuring sector priority (e.g. domestic water satisfied first, then livestock and so on).

I also finished coding the coupler, mediator and routing part (we use MOSART at the moment).

The part related to CTSM was already tested and seems to work correctly.
On the other side, I got stuck on a bug when I try to transfer information (withdrawal and return flow fluxes to the routing through the coupler).

The error I get (on the first step):
ERROR: One or more of the output from CLM to the coupler are NaN

The log file can be accessed here : /glade/scratch/staranu/test7_sectorWater_run1/run/cesm.log.4710503.chadmin1.ib0.cheyenne.ucar.edu.220621-063059

And the code used to do the run is:
/glade/u/home/staranu/CTSM_all_versions/test7_sectorWater

Here is what I already tried:

1. I looked into the log file, and it seems that the error happens for the first in order flux related to my implementation which I try to transfer (Flrl_dom_withd). I thought that maybe the values which are provided by the land part is the issue. This is why I changed the value for waterlnd2atmbulk_inst%qdom_withd_grc(:) with a small constant (it is this qdom_withd_grc from water instance object which is transferred to the coupler). But this didn't helped, as the ERROR persisted and didn't moved to the next field. This means that the problem should be somewhere higher (coupler/mediator or routing).

2. Since the problem is happening at the first step, I think there is nothing to look at in the routing yet. So I checked the code for the coupler and mediator part. But I couldn't really find anything suspicious. One thing I thought is strange is that in the ERROR messages in the log file, for each batch analyzed not all values are NaNs, and in some batches there can be more or less NaNs values. This made me think that maybe this is somehow related to the process of normalization/remapping which I apply in the coupler and mediator part (e.g. some division by 0)? But again, I didn't find anything suspicious, and the code I wrote is based on what was already in use for the irrigation part.

At the moment I am dry of other ideas, so I would like to ask for some advice. How would you approach this bug? What you would look into? etc...

Cheers,
Sabin

[ekluzek]

Hi Sabin. Hmmm. I recommend to try some of the normal type of things that are done for debugging. You'll see this in the trouble shooting section of the user's guide.

But the first thing to try is to clean the build and recompile after setting DEBUG=TRUE. That might abort earlier and give you a better traceback. It's also good to try this on a smaller problem as small as possible. So trying a single point case would be good to setup.

The other thing is to make sure a vanilla CLM case works when you turn off your new features.

The thing I wonder is if your new data isn't being set or is still set to NaN at the beginning and is affecting other variables.

So those are a few things to try.

[TaranuDev]

Hi!

We have been working for some time now on implementing human sectoral water abstractions in the CESM model.
This includes abstractions for domestic, livestock, thermoelectric, manufacturing and mining sectors.
At the moment the modeling, as well as testing and validation, is completed. So we are ready to deploy the solution to the main development branch.

The repositories which will be affected are:
https://github.com/ESCOMP/CTSM
https://github.com/ESCOMP/MOSART
https://github.com/ESCOMP/CMEPS
https://github.com/ESCOMP/CESM_CPL7andDataComps

From NCAR, our main coordinators for this project are David Lawrence and Erik Kluzek.
Other people from NCAR who contributed to the discussions are: Sean Swenson, Naoki Mizukami, and Andrew Wood.

Outside the NCAR, the people involved or supporting this project are: Wim Thiery, Yoshihide Wada, Ting Tang, Yi Yao, Inne Vanderkelen, Steven Johan De Hertog.

For any questions, feel free to contact me at sabin.taranu@vub.be.

Best regards,
Sabin