Check the values plausibility if known = physical plausible limits (V452)

[JavierDiezSierra]

[JavierDiezSierra]

Hi @Ayoubnac1, @glevava,

I’ve just submitted a pull request with the first version of the "detect_physically_impossible_outlier.py" check.

The check itself is quite simple: it searches for variables whose standard_name is listed in the "outliers_thresholds.json", which contains plausible physical ranges for each variable (I'm not sure where this resource file should be, please move it to the appropriate place). The script detects whether a climate variable contains physically impossible values based on those ranges. It also provides extra information such as the number of time steps with outliers, the number of outliers per time step, and more. I think It would be interesting also provide the values of the outliers and their position (time, lon, lat, etc.) in a .txt with a limit of lines in case there are too much outliers to avoid generating large output files.

Have you thought about where the output logs should be saved?

I also included some .nc files for unit testing.

I have a question regarding the parallelization/chunking strategy you plan to follow. We usually work with xarray and dask, but in this case we preferred to use netCDF4 to stay aligned with your code base and to have more low-level control if needed. It could happen that opening the entire file with netCDF4 to detect outliers leads to memory issues. Should we handle chunking and parallelization internally ourselves, or do you already have a file-level strategy in mind for this?

It still needs improvements — currently, it's set up for input datasets with dimensions (time, x, y), but we’ll generalize it soon to support datasets with only (x, y) (e.g., static variables) or an additional member dimension (as in the Atlas). Over the coming days, we’ll work on generalizing the function and begin populating the JSON file with physically plausible ranges.

An additional related test is one aimed at detecting data quality issues based on statistics. In this case, the range for each variable will be calculated from the input dataset (per grid cell), using a number of standard deviations.

Here’s a simple test output example:

(qaqc) javi@aura:~/Projects/C3S2_382/github/esgf-qc-init_data_plausibility_check$ pytest -vs compliance_checker/tests/test_checks/data_plausibility_checks/test_detect_physically_impossible_outlier.py 
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.13.3, pytest-8.3.5, pluggy-1.5.0 -- /home/javi/mambaforge/envs/qaqc/bin/python3.13
cachedir: .pytest_cache
rootdir: /home/javi/Projects/C3S2_382/github/esgf-qc-init_data_plausibility_check
configfile: pyproject.toml
plugins: requests-mock-1.12.1, vcr-1.0.2, time-machine-2.16.0
collecting ... ncgen: /home/javi/Projects/C3S2_382/github/esgf-qc-init_data_plausibility_check/compliance_checker/tests/data/non-comp/badname.netcdf line 1: syntax error, unexpected end of file, expecting NETCDF
collected 2 items                                                                                                                                                                                          

compliance_checker/tests/test_checks/data_plausibility_checks/test_detect_physically_impossible_outlier.py::TestPhysOutliers::test_detect_physically_impossible_outlier No outliers detected for variable 'tas'.
[Check for outliers in a dataset based on predefined thresholds. (*2): (1, 1)]
PASSED
compliance_checker/tests/test_checks/data_plausibility_checks/test_detect_physically_impossible_outlier.py::TestPhysOutliers::test_detect_physically_impossible_outlier_fails Outliers detected for variable 'tas'.
[Check for outliers in a dataset based on predefined thresholds. (*2): (0, 1) (Physically impossible outliers detected in the dataset.Example, value:69576296.0,coordinates:[1.82825000e+04 6.76499987e+00 1.51850004e+01] for a total of 3 outliers and thresholds parameters:{'min': 0, 'max': 400, 'unit': 'Kelvin', 'projects': ['CORDEX', 'CMIP', 'C3SATLAS']})]
PASSED

[Ayoubnac1]

Hi @JavierDiezSierra ,

Thanks so much for your work on the detect_physically_impossible_outlier.py check

To enhance this for different WCRP projects, here are a couple of suggestions for your check

**Project-Specific Threshold Files:** 

Instead of a single JSON file with a "projects" key inside, we're thinking it would be cleaner to have separate JSON threshold files for each project (e.g., cmip6_thresholds.json, cordex_thresholds.json). This would simplify management.

Flexibility in Variable Targeting within Project Thresholds:

 Inside a project-specific threshold file (e.g., cmip6_thresholds.json), standard_name remains the ideal primary key for variables. However, for cases where a dataset might lack a standard_name, it would be helpful if the WCRP plugin could explicitly tell your check which variable name to use.

Detailed Outlier Output:
The .txt file for detailed outlier info is a good idea for a supplementary output.

Memory/Performance:
We'll monitor this and can discuss options like internal chunking if needed.

Generalization & Statistical Check:
Your plans for generalizing dimensions and adding a statistical outlier check sound like great future steps!

[chantreux]

Hi @Ayoubnac1,
We are wondering with @JavierDiezSierra how to run ESGF-QC on a full dataset for data testing. We have tested our functions on individual files so far, but now we would like to test them on an entire dataset. The files are organized annually. Could you clarify how the workflow is designed to handle multiple input files in this scenario?

Specifically, we’d like to know:

Does the workflow automatically concatenate the files if a list of files is given as an input, into a single dataset for analysis, or should we pre-process them in a netcdf single file before running the tool?

Thanks for your guidance!

[sol1105]

Hi @chantreux ,

there is no method in esgf-qc to run checks over multiple files and aggregate / summarize the results. For a different compliance_checker plugin I set up the following tool for this purpose:
https://gitlab.dkrz.de/udag/cc-qa
I now added support for esgf-qc, so if you want you can give it a try. It is yet a prototype and I am happy for any feedback.

We were discussing this briefly in the last dev. meeting (there is also #15):
While there was no decision made yet, we might adopt and further develop cc_qa for running the QA-workflow.

To run it with the wcrp_cmip6 checker you would have to run:

ccqa -t "wcrp_cmip6:latest" -o /path/to/result_dir /path/to/datasets/to/check

I also included instructions in the repository readme to install esgf-qc as a compliance_checker plugin rather than to replace it, in case you want to have both tools installed next to each other (esgf-qc overwrites a possible compliance_checker installation the way it is currently set up).

[chantreux]

Hi @sol1105 ,

Thank you very much for the clarification and for sharing the cc-qa tool! I think we will give it a try as soon as we can and share our feedback with you. If I have more questions or feedback I will directly write in #15.