Consider reworking how truncation of small values is done

[billsacks]

Emerging from doing more thinking about #2458 -

I realized a possible issue with relaxing the tolerance too much – or with how this truncation is done in general – and this led me to think of a different, probably more robust way to handle this truncation:

The issue is that, in many of its uses, I think it will be a problem if we truncate the bulk to 0 while not truncating one of the tracers, or vice versa. In fact, I think that's the problem I was trying to prevent with this truncate_small_values, and it should prevent the issue much more often than it causes issues, but there is a potential for it to (rarely) cause issues like this – for example, if the tolerance is 1e-12 and the bulk is reduced by a factor of 9.9e-13 while a tracer is reduced by a factor of 1.1e-12, the bulk would be truncated to 0 while the tracer is not. I'm not sure if relaxing the tolerance would lead this to be more likely, but it seems like it might be.

This led me to think about a completely different way to handle this: Rather than doing this truncation after each state update independently on bulk and tracer states (as is currently done), instead do this all at once – probably near the end of the time step – in a way that is coordinated between the bulk and tracer variables. There is already infrastructure in place for looping over all water variables that have tracer quantities (e.g., for the sake of TracerConsistencyCheck), so we could use that infrastructure for this purpose.

I'm envisioning something like this:

1. Add a flag to each water variable saying whether this variable should be involved in the truncation (typically true for state variables, false for others).
2. (Optional, if needed) Add an optional custom_rel_epsilon for each water variable to override the default value for truncation for that variable (this may be less needed with this scheme than it is for the current scheme)
3. For the variables where this is needed, add a second copy that stores the value at the beginning of the time step; this could be done by the infrastructure that allocates the water variables that have tracers (that's how I'd lean right now, without having looked at what that would involve), or could be done by having a second copy of WaterStateType (and maybe one or two others) if all of the relevant variables are there.
4. At the beginning of the time step, call a routine that saves a copy of the beginning of the time step values
5. At the end of the time step, call a routine that truncates variables that have gone to near-zero.

More details on the actual truncation: I envision this would be done by going through each variable that has the truncation flag. If the bulk has gone to near-zero (defined by a relative epsilon, relative to the value at the start of the time step), then we would:

• Set the bulk to exactly 0
• Loop through all tracers for this variable
  • If the tracer has gone somewhat close to 0, truncate it to 0. Importantly, note that "somewhat close" here would be defined in a more relaxed way than the epsilon used for the bulk – e.g., allow a few extra orders of magnitude. e.g., if the rel epsilon for the bulk is 1e-12 (12 orders of magnitude decrease), we would consider that things are consistent if the tracer has gone down by at least, say, 1e-9 (9 orders of magnitude).
  • If the tracer is still significantly non-zero, abort with an error

I'm not sure off-hand what to do if a tracer variable has gone to near-zero without the corresponding bulk going to near-zero. I think we don't want the addition of tracers to affect the evolution of the bulk quantities, so that makes me think that we should not truncate the bulk in this situation. I think it would be okay to truncate the tracer to zero while leaving the bulk non-zero: after all, the code should work when we have exactly zero tracers of a given tracer.

One issue with this algorithm is that we could still have issues of the bulk being zero and a tracer being non-zero (or vice versa) in the middle of the time step, after a partial state update. To be safe, we may need to call this truncation code more frequently - e.g., as frequently as we currently do the TracerConsistencyCheck - which would be wasteful if we're looping over all variables each time. We have discussed a long-term vision of trying to consolidate the hydrology state updates into fewer places, which would improve this situation, but until then we may need to either (a) call this truncation code multiple times in the time step, or (b) reduce the calls to TracerConsistencyCheck and make sure that the code in the middle of the time step isn't sensitive to one of the bulk or tracer being zero while the other is non-zero (I'm not sure if this will be possible). With (a) we could theoretically have the truncation just be done for explicitly specified variables rather than looping over all hydrology state variables, but it feels like that could get messy and error-prone.