Strategy for Multi-File Results Aggregation and Summarization

[Ayoubnac1]

Question from Martin Schupfner :

How to aggregate / summarize test results from multiple files?
compliance-checker tests file-by-file without any option to summarize
test results in a user-friendly way (afaik). Do you already have a plan
how to deal with this? Would you implement it directly into the
compliance-checker codebase? I am currently working on a prototype for a
wrapper, submitting the cc-checks for each file and then collecting and
summarizing the results.

[Ayoubnac1]

• Compliance-checker currently processes file-by-file, We don't plan to modify the core compliance-checker codebase to add this aggregation directly. We want to keep its focus on single-file checking to maintain its original design philosophy.

• We also plan to leverage different output formats or verbosity levels within compliance-checker itself (as it already supports, e.g., JSON, text, html ...). Users could choose a verbose mode (all check results and informations, pass or fail) or a summary mode (e.g., only failed checks), but the aggregation across files would be a separate step.

• Our approach aligns with yours, an external wrapper or a post-processing step to aggregate the report .We are very interested in your prototype wrapper.

[sol1105]

The prototype was set up for CORDEX-CMIP6 (so to run the cc-plugin-cc6 and cf checks) but could be extended for esgf-qc or any compliance-checker based application. It is available here: https://gitlab.dkrz.de/udag/cc-qa
It also includes continuity and consistency checks, but more on that in #16.
cc-qa runs the cc checks for each netCDF file in the selected folder and groups them by dataset id. It also clusters the error messages and outputs a full and a clustered summary as json.

I set up a separate repo to host the (clustered) QA results (https://gitlab.dkrz.de/udag/qa-results) and a small web app that can display the results (to be selected from disk or from the QA results repo): https://cmiphub.cloud.dkrz.de/info/display_qc_results.html