`NaN` in metrics

[tomsail]

@JackReevesEyre-NOAA reported the problem of having NaNs for general metrics (I'd guess cr principally) when using higher quantile values in get_stats().

There is a confusion on the default quantile used (currently 0).

Although there is a Usage example at the end of the README:

from seastats import get_stats, GENERAL_METRICS_ALL, STORM_METRICS_ALL
general = get_stats(sim, obs, metrics = GENERAL_METRICS)
storm = get_stats(sim, obs, quantile = 0.99, metrics = STORM_METRICS) # we use a different quantile for PoT selection
pd.DataFrame(dict(general, **storm), index=['abed'])

it'd be less confusing:

1. to add a DISCLAIMER warning users at the beginning,
2. or even spit the current get_stats() in 2 separate API calls
   • one for storms
   • one for general metrics

[JackReevesEyre-NOAA]

Thanks for documenting this, @tomsail. I've changed my usage to be more like the README, along the lines of:

general = get_stats(sim, obs, metrics = GENERAL_METRICS)
storm = get_stats(sim, obs, quantile = <user_interface_value>,  metrics = STORM_METRICS)

[JackReevesEyre-NOAA]

One other question while I've been looking into this: the storms.match_extremes() function defines error as the absolute value of difference (df["error"] = abs(df["diff"])). This is then carried through to the seastats R1, R3, and error metrics. Is this the expected behavior too? My intuition is that "error" does not automatically mean absolute value. But that could be my ignorance!

[tomsail]

Hi Jack, thank for catching this too.

Indeed I have chosen:

• diff for the difference between modeled and observed values
• error for the absolute value of the diff
  Which I realize now might be confusing names and do not follow any convention.

The following names would be indeed more intuitive and scientifically thorough:

• diff → signed_error or error (bias being already taken)
• error → abs_error or absolute_error

Let me know what you think

[JackReevesEyre-NOAA]

Hi @tomsail -- thanks for explaining.

• I would say that diff is fine (and error would be fine as an alternative) for the signed value.
• Either abs_error or absolute_error are good for the absolute value metric. I slightly prefer absolute_error. And if we need to differentiate between the time series and the mean, then mean_absolute_error or MAE could be used for the mean.

[JackReevesEyre-NOAA]

Thanks for documenting this, @tomsail. I've changed my usage to be more like the README, along the lines of:

general = get_stats(sim, obs, metrics = GENERAL_METRICS)
storm = get_stats(sim, obs, quantile = <user_interface_value>,  metrics = STORM_METRICS)

This usage does now seem to be occasionally raising an Exception at the following line
https://github.com/oceanmodeling/seastats/blob/c3e538e9df770ed81797dbbff543802b92f495b5/seastats/__init__.py#L200C31-L200C58
where the issue is calling
extreme_df["error"].iloc[0] with an empty data frame.

It raises an exception like

...
in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds

I think that replacing the lines like extreme_df["error"].iloc[0] with extreme_df["error"].iloc[:0] might fix the issue. It would then return an empty series or nan value in cases where extreme_df has zero rows of data.