Skip to content

Conversation

bartgol
Copy link
Contributor

@bartgol bartgol commented Jul 28, 2025

Changes:

  • For floating point numbers, ensure it is ALWAYS the same number.
  • Switch to an inline templated constexpr variable.
  • Test behavior of fill_aware_combine
  • Test behavior of field ops with a fill-value RHS

[non-BFB]


Note: I marked this as non-BFB, as it can technically change output files. However, the changes should only affect output. Although processes that use input files that contain fill-value entries or need vertical remap may be affected.

Fixes #7548
Closes #7550

Edit: still need to address something related to packs.

@bartgol bartgol requested a review from mahf708 July 28, 2025 19:13
@bartgol bartgol self-assigned this Jul 28, 2025
@bartgol bartgol added bug fix PR non-BFB PR makes roundoff changes to answers. EAMxx Issues related to EAMxx labels Jul 28, 2025
@bartgol bartgol force-pushed the bartgol/eamxx/fix-fill-val-def branch from a352ea5 to 2a3aeda Compare July 28, 2025 19:15
For floating point numbers, ensure it is ALWAYS the same number.
Also, switch to an inline templated constexpr variable.
@bartgol bartgol force-pushed the bartgol/eamxx/fix-fill-val-def branch from 2a3aeda to 54c94bb Compare July 28, 2025 19:16
@bartgol bartgol marked this pull request as draft July 28, 2025 19:18
@bartgol bartgol force-pushed the bartgol/eamxx/fix-fill-val-def branch from 54c94bb to ed474f2 Compare July 28, 2025 21:05
@bartgol bartgol marked this pull request as ready for review July 28, 2025 21:05
auto fv_d = fill_value<double>;
auto fv_f = fill_value<float>;

REQUIRE (fv_d==fv_f);
Copy link
Contributor

@mahf708 mahf708 Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would all things below still yield the same result?

  • Real(fill_value<double>)
  • static_cast<float>(fill_value<double>)
  • static_cast<double>(fill_value<float>)

If not, should we leave some notes on which inside of EAMxx is the ultimate fill_value so that people can follow instructions to retrieve it?

Copy link
Contributor Author

@bartgol bartgol Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Casting to Real should yield the same number, since fv_d==fv_f, and Real is either double or float. I am not sure what the 2nd and 3rd bullet referred to, as you probably forgot to put the type you cast to. But I assume you prob want to know whether, given fv_d and fv_f above, we have fv_d == fv_d == fill_value<Real> == static_cast<Real>(fv_d) == static_cast<Real>(fv_f) == static_cast<double>(fv_f) == static_cast<float>(fv_d) and so on, and the answer is "yes". They are ALL the same numerical value. It's even safe to do

double x = ...;
auto fv = fill_value<float>;
if (x==fv) 

since fv will be promoted to double without changing the numerical value.

@bartgol
Copy link
Contributor Author

bartgol commented Jul 29, 2025

Will investigate the fails tomorrow.

@mahf708
Copy link
Contributor

mahf708 commented Jul 29, 2025

@bartgol do you think we should take out optionality to set fill_value at all? Like EAMxx has only one fill_value, live it with? I am having a pretty nasty bug in #7551 where the fill val in field_utils_impl is somehow 0 👀 , so gonna hardcode it!

@bartgol
Copy link
Contributor Author

bartgol commented Jul 29, 2025

@bartgol do you think we should take out optionality to set fill_value at all? Like EAMxx has only one fill_value, live it with? I am having a pretty nasty bug in #7551 where the fill val in field_utils_impl is somehow 0 👀 , so gonna hardcode it!

That's what #7537 would do. But I factored out this bug fix in its own PR, as it seemed orthogonal. Once this is merged, I'll focus on that one.

bartgol added 3 commits July 29, 2025 12:41
Fill value represent new data to ignore, so its type
must match that of the RHS
Avoid (tiny) where_expression's overhead for simple builtin types
@bartgol bartgol requested a review from mahf708 July 29, 2025 21:47
Copy link
Contributor

@mahf708 mahf708 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

Maybe revisit PR description to ensure it is accurate and this looks like it will end up being BFB? If only diagnostic stuff is DIFFing, then technically it is BFB, right?

@bartgol
Copy link
Contributor Author

bartgol commented Jul 29, 2025

Looks good!

Maybe revisit PR description to ensure it is accurate and this looks like it will end up being BFB? If only diagnostic stuff is DIFFing, then technically it is BFB, right?

Well, I wanted to be safe, and not mark it as BFB. Not that we test it, but nudging that reads in data may use fill value to pad missing data (e.g., input TOA is lower than the model grid's one).

bartgol added a commit that referenced this pull request Jul 29, 2025
Changes:

- For floating point numbers, ensure it is ALWAYS the same number.
- Switch to an inline templated constexpr variable.
- Test behavior of fill_aware_combine
- Test behavior of field ops with a fill-value RHS

[non-BFB for certain EAMxx outputs]
@bartgol bartgol merged commit 982878e into master Jul 29, 2025
18 checks passed
@bartgol bartgol deleted the bartgol/eamxx/fix-fill-val-def branch July 29, 2025 22:55
@rljacob
Copy link
Member

rljacob commented Jul 30, 2025

What is the fillvalue value? In shr_cont_mod.F90, its 1.0e30_R8

@mahf708
Copy link
Contributor

mahf708 commented Jul 30, 2025

What is the fillvalue value? In shr_cont_mod.F90, its 1.0e30_R8

there are three types:

constexpr T fill_value =
  std::is_integral_v<T> ? std::numeric_limits<int>::max() / 2
                        : std::is_floating_point_v<T> ? std::numeric_limits<float>::max() / static_cast<float>(1e5)
                                                      : std::numeric_limits<char>::max();

The float one you're likely interested in ... is something like of these two (which is 3.4e33 ish)
3402823466385288812099273089875968 3402823583246828306656939785322496

@bartgol
Copy link
Contributor Author

bartgol commented Jul 30, 2025

@rljacob Is the numerical value important? The variables in the nc file will contain the _FillValue attribute, which is what the postproc tools use to detect where the data is missing. In this framework, the actual value of that seems irrelevant, no?

@rljacob
Copy link
Member

rljacob commented Jul 30, 2025

Right it shouldn't matter. I think CMIP used to define a standard of 1e+20 which is useful for tools that don't know to check the FillValue attribute or data sets that don't set that attribute (but do put a fillvalue in the data). Lets not worry about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug fix PR EAMxx Issues related to EAMxx non-BFB PR makes roundoff changes to answers.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EAMxx: fill_aware_combine bug
3 participants