Support for mixed precip/noprecip input and AR order 1 by ladc · Pull Request #544 · pySTEPS/pysteps

ladc · 2026-03-27T07:40:29Z

Fix several issues in case of intermittent rainfall in the radar images:

Works towards fixing issue #543.

Reduce overall number of tests and refactor code Add dedicated AR order parameterization

- similar to the tests in test_blending_steps.py

…meters inherited from blending test

…cript.

codecov · 2026-04-02T16:25:15Z

Codecov Report

❌ Patch coverage is 94.14894% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.20%. Comparing base (29a317b) to head (15caf3a).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
pysteps/utils/check_norain.py	77.27%	5 Missing ⚠️
pysteps/blending/skill_scores.py	70.00%	3 Missing ⚠️
pysteps/blending/steps.py	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #544      +/-   ##
==========================================
+ Coverage   83.89%   84.20%   +0.31%     
==========================================
  Files         168      170       +2     
  Lines       14678    14974     +296     
==========================================
+ Hits        12314    12609     +295     
- Misses       2364     2365       +1

Flag	Coverage Δ
unit_tests	`84.20% <94.14%> (+0.31%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

FelixE91 · 2026-04-07T10:52:13Z

We consider these fixes important enough to merge them into the master branch. They can have a direct positive impact on the operation of pysteps, as they
(i) prevent some runs from crashing when mixed time steps with and without precipitation are present in the radar inputs
(ii) provide more readable and clearer error messages if the run does abort due to mixed time steps with and without precipitation in the radar inputs
The tests (pytest) are still generating numerous warnings. Most of them are unrelated to the code we added here.
The warnings from the pytest execution should be investigated as a separate issue #545

FelixE91 · 2026-04-07T13:41:20Z

The implemented solution prevents pysteps from crashing when the radar input data contains both precip and noprecip time steps.
If possible, the most recent time steps with precipitation are used, and the ar_order value is adjusted (i.e. reduced) as needed.
Otherwise, the radar input is set to an observation array without precipitation and a warning is printed. Pysteps with NWP blending will still run if rain is present in the NWP.

pysteps/blending/steps.py

mats-knmi · 2026-04-08T06:30:45Z

pysteps/blending/steps.py

+        )
+
+    # Check all time steps for zero-precip (constant field, minimum==maximum)
+    zero_precip = [np.nanmin(obs) == np.nanmax(obs) for obs in precip]


Why do you not use the check_norain method here?

The output differs. While check_norain returns a Boolean value for the entire array, the check here is performed for each individual timestep and returns a list of length ar_order+1. This is a simple technical check to verify whether the input array is constant for the respective timestep, which must be avoided in order to calculate the spatial correlation coefficient.
Note that check_norain is still used later in steps.py to apply the specified thresholds.

If you prefer, we could call this const_precip or so to avoid confusion?

pysteps/nowcasts/steps.py

mats-knmi · 2026-04-08T06:44:23Z

We have been having these issues as well in our processing, so I hope this PR finally fixes it. To me it looks good, I just had a couple of questions. If they can be answered I can approve it.

FelixE91

I have tried to answer @mats-knmi's questions. Final approval for the updates of the blending and autoregression functions is still pending, most of it to enable AR-1 runs.

RubenImhoff · 2026-04-09T07:52:42Z

Thanks for picking this up, great work! I will go through the changes and leave my review as well.

Regarding AR_order = 1, indeed this was not yet supported (also mentioned in the code by the way: "The order of the autoregressive model to use. Must be >= 1."). I am glad to see that we are fixing this now, too. :)

RubenImhoff · 2026-04-09T08:12:56Z

pysteps/utils/check_norain.py

+        )
+
+    # Check all time steps for zero-precip/constant field (constant field, minimum==maximum)
+    const_precip = [np.nanmin(obs) == np.nanmax(obs) for obs in precip]


A lot of 'no_rain' cases consist of a couple of cells containing clutter. Our experience is that the autoregressive model and/or determining the correlation coefficients then also fail. So, ideally we check for that, too.

In the check_norain function, we have built in a norain_thr, which is the minimum fraction of rainy pixels, and a threshold (precip_thr). I can imagine that it would be good, and consistent, to use that here, too. If so, the function becomes more similar to the check_norain function, so we could even see if an integration of the two functions makes more sense at that point (no preference on my side).

Ah, gotcha! That makes sense, now I understand why the check_norain method is for flexible than just checking the min and max.
As mentioned, it was a technical test here to avoid const arrays as they cause errors in the correlation coefficients.
I will modify the function to use the thresholds and check for a constant field.

Great, thanks, @FelixE91!

Also in check_norain we now also have support for a window function. I am not sure how it works in this case, but we have had instances where there was precipitation outside the window function, so it passed a no_rain check, but after the window function was applied it was still a constant field so we got an error.

I am not sure if that is the case here, but imo just using the check_norain method here is slightly more robust and it should also cover the case of a constant field that you are checking for.

Do you have a specific reason why using check_norain here would be worse than what you are doing now?

I mean, I guess we can assume that a constant field exists if all precip is zero only. It woudl be very unlikely to have the same non-zero value at every pixel ;)
Since I need a list of length ar_order+1 as output, I can simply do const_precip = [check_norain(obs) for obs in precip] that should work and address @RubenImhoff 's comment, too.

pysteps/blending/steps.py

FelixE91 · 2026-04-09T14:02:00Z

pysteps/blending/steps.py

+            "precip_thr": precip_thr,
+            "norain_thr": norain_thr,
+            # Set default window function to "tukey" to match the default in check_inputs() below
+            "win_fun": "tukey" if noise_kwargs is None else noise_kwargs["win_fun"],


I was not totally sure whether this check should be for the entire domain or for the window function that is used in later noise generation.
Here I set it to match this window function ("tukey" by default).
If you think it is better to check the full domain at this point of the code, we simply delete lines 3665/3666.

FelixE91 · 2026-04-09T14:02:54Z

pysteps/nowcasts/steps.py

+            "precip_thr": precip_thr,
+            "norain_thr": norain_thr,
+            # Set default window function to "tukey" to match the default in check_inputs() below
+            "win_fun": "tukey" if noise_kwargs is None else noise_kwargs["win_fun"],


Same as in blending.steps.py.
If we want to check the entire domain here, we can delete lines 1515/1516.

FelixE91 added 3 commits March 26, 2026 18:01

Fix precip-noprecip input array blending failure

cbd0d06

Add tests for blending with noprecip and precip input time steps

4d01c01

fix prev in check_previous_radar_obs()

3948d5d

ladc linked an issue Mar 27, 2026 that may be closed by this pull request

Precip + no-precip input time steps cause blending failure AND in general ar_order=1 causes blending failure #543

Open

Improve tests for blended forecast

5170768

Reduce overall number of tests and refactor code Add dedicated AR order parameterization

ladc force-pushed the precip-noprecip_input_fix_and_ar_order-1_fix_issue543 branch from 2e24248 to 5170768 Compare March 27, 2026 12:44

ladc and others added 11 commits March 27, 2026 13:46

Fix formatting in blending/steps.py

f3355c1

fix ar_order=1 runs (AR-1 model)

4650d8b

add warning message for ar_order 1

bd7f399

Fix precip-noprecip input array nowcast failure

b49902e

fix mixed precip blending test

9135618

Add new nowcast test for mixed precip-no-precip input

0f45439

- similar to the tests in test_blending_steps.py

improve warning messages in check_previous_radar_obs

3c723ce

fix renormalize cascade

7d3a2df

Fix n_ens_members to 1 in blending tests and update nowcast test para…

2cb883f

…meters inherited from blending test

Remove more leftovers from blending from the test_nowcasts_steps.py s…

26fa133

…cript.

reformatted to match default

e605150

FelixE91 added 6 commits April 3, 2026 09:18

black reformatting

c47d49d

black reformatted

3188d38

fix codacy issue

1e05808

add bandit config to exclude assert check in tests

2cc64a4

Improve code readability

12d10a5

black and codacy checks

55261f1

FelixE91 added 2 commits April 7, 2026 15:25

add warning message

4e0b231

fix codacy f-string

2bcc065

FelixE91 marked this pull request as ready for review April 7, 2026 13:56

dnerini requested a review from RubenImhoff April 7, 2026 20:08

dnerini self-requested a review April 7, 2026 20:08

mats-knmi reviewed Apr 8, 2026

View reviewed changes

pysteps/blending/steps.py Show resolved Hide resolved

mats-knmi reviewed Apr 8, 2026

View reviewed changes

pysteps/nowcasts/steps.py Outdated Show resolved Hide resolved

FelixE91 added 2 commits April 8, 2026 12:13

Fix naming

f8602b5

black check

62f03cc

FelixE91 reviewed Apr 8, 2026

View reviewed changes

RubenImhoff requested changes Apr 9, 2026

View reviewed changes

use check_norain in check_previous_radar_obs

ff2bc18

FelixE91 reviewed Apr 9, 2026

View reviewed changes

fix codacy issue

15caf3a

Conversation

ladc commented Mar 27, 2026 • edited by FelixE91 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

FelixE91 commented Apr 7, 2026

Uh oh!

FelixE91 commented Apr 7, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FelixE91 Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mats-knmi commented Apr 8, 2026

Uh oh!

FelixE91 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RubenImhoff commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FelixE91 Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ladc commented Mar 27, 2026 •

edited by FelixE91

Loading

codecov bot commented Apr 2, 2026 •

edited

Loading

FelixE91 Apr 8, 2026 •

edited

Loading

FelixE91 left a comment •

edited

Loading

FelixE91 Apr 9, 2026 •

edited

Loading