feat: change default behavior of fail_on_errors and add roadmap and demo suite

Author: Michael Hallik

CHANGELOG.md

@@ -12,4 +12,62 @@ All notable changes to the project will be documented in this file.
 - Use Poetry for dependency and release management.
 
 ## [1.0.1] - 2025-06-09
-- Refactor: simplify the __init_() method of class XmlValidator
+- Refactor: simplify the __init_() method of class XmlValidator
+
+## [2.0.0] - 2025-06-14
+
+### Breaking changes
+
+- Changed default behavior of the `fail_on_errors` setting.
+  - When using the `Validate Xml Files` keyword, test cases that detect validation errors (of any kind) will have FAIL status.
+  - Previously, test cases received a PASS status, even when validation errors were found, unless `fail_on_errors=True` was explicitly set.
+  - This change improves alignment with common validation expectations and was motivated by user feedback.
+  - To preserve the old behavior, explicitly set `fail_on_errors=False` either during library import or per keyword call:
+  
+    ```robot
+    Library    XmlValidator    fail_on_errors=False
+    ```
+
+    Or:
+
+    ```robot
+    Validate Xml Files    path/to/file.xml    fail_on_errors=False
+    ```
+
+Note:
+
+The library's batch validation behavior remains unchanged. That is, `fail_on_errors=True` does *not* short-circuit the validation process in any way.
+
+For example
+- If you validate fifteen XML files and five of them contain schema violations or other errors, all files will still be processed.
+- Errors are simply collected throughout the run and reported collectively, only after the final file has been (fully) processed.
+- The test case will fail (assuming `fail_on_errors=True`) only after all files have been checked, ensuring comprehensive diagnostics.
+
+### Added
+
+- Support for setting `fail_on_errors` also at the *library import level*
+  - Applies to all calls to `Validate Xml Files` unless explicitly overridden during keyword call (see also earlier/above).
+  - Improves usability and test readability.
+
+- New Robot Framework demo test suite containing 8 self-contained test cases.
+  - Demonstrates key features:
+    - Single and batch XML validation
+    - Schema matching by filename and namespace
+    - Custom error facets
+    - Malformed XML handling
+    - XSD includes/imports
+    - CSV export
+  - Serves as a quickstart reference for users exploring the library.
+
+- `ROADMAP.md` to document planned features and/or improvements.
+
+### Changed
+
+- Integration test suite updated to support and validate the new `fail_on_errors` behavior.
+  - Tests now reflect expected `FAIL` or `PASS` statuses depending on configuration.
+  - Verifies both default (library-level) and keyword-level overrides.
+- Minor changes in the Robot Framework logging (based on user feedback).
+  - Changed some from level=WARN to level=INFO.
+  - Some got commented out.
+  - Some indentation changes as well.
+- Changed documentation (READEME.md, keyword documentation, etc.) to reflect additions and changes in this version.

README.md

@@ -24,19 +24,26 @@
     - [Examples](#examples)
       - [Using a preloaded schema](#using-a-preloaded-schema)
       - [Defer schema loading to the test case(s)](#defer-schema-loading-to-the-test-cases)
-      - [Importing with preloaded XSD that requires a base\_url](#importing-with-preloaded-xsd-that-requires-a-base_url)
-      - [Importing with custom error\_facets](#importing-with-custom-error_facets)
+      - [Importing with preloaded XSD that requires a `base_url`](#importing-with-preloaded-xsd-that-requires-a-base_url)
+      - [Importing with custom `error_facets`](#importing-with-custom-error_facets)
+      - [Importing with \`fail\_on\_errors=True](#importing-with-fail_on_errorstrue)
       - [Further examples](#further-examples)
   - [Using the library](#using-the-library)
     - [Keyword overview](#keyword-overview)
-    - [Error collection](#error-collection)
+      - [Error collection](#error-collection)
+      - [Batch mode](#batch-mode)
+      - [Single file mode](#single-file-mode)
+      - [Test case status - fail\_on\_error](#test-case-status---fail_on_error)
+      - [Dyanmic XSD resolution](#dyanmic-xsd-resolution)
+    - [Error collection](#error-collection-1)
       - [XSD Schema violations.](#xsd-schema-violations)
       - [Malformed XML](#malformed-xml)
       - [File-level issues](#file-level-issues)
       - [Final note on error collection](#final-note-on-error-collection)
     - [Keyword documentation](#keyword-documentation)
     - [Keyword example usage](#keyword-example-usage)
       - [A few basic examples](#a-few-basic-examples)
+      - [Demo test suite file as examples](#demo-test-suite-file-as-examples)
       - [Integration tests as examples](#integration-tests-as-examples)
     - [Example console output](#example-console-output)
     - [Example CSV output](#example-csv-output)
@@ -58,6 +65,8 @@
     - [Continuous Integration \& GitHub templates](#continuous-integration--github-templates)
   - [Class architecture (Simplified)](#class-architecture-simplified)
   - [Project Structure](#project-structure)
+  - [Changelog](#changelog)
+  - [Roadmap](#roadmap)
   - [License](#license)
   - [Author](#author)
 
@@ -130,11 +139,12 @@ See the [Robot Framework Library Scope docs](https://robotframework.org/robotfra
 
 ### Library arguments
 
-| Argument       | Type        | Required? | Description                                                                        |
-|----------------|-------------|-----------|------------------------------------------------------------------------------------|
-| `xsd_path`     | `str`       | No        | Path to an XSD file or folder to preload during initialization. In case of a folder, the folder must hold one file only.                                                                                                             |
-| `base_url`     | `str`       | No        | Base path used to resolve includes/imports within the provided XSD schema.         |
-| `error_facets` | `list[str]` | No        | The attributes of validation errors to collect and report (e.g., `path`, `reason`) |
+| Argument        | Type        | Required? | Description                                                                                                            | Default         |
+|-----------------|-------------|-----------|------------------------------------------------------------------------------------------------------------------------|-----------------|
+| `xsd_path`      | `str`       | No        | Path to an XSD file/folder to preload during initialization. In case of a folder, the folder must hold one file only.  | None            |
+| `base_url`      | `str`       | No        | Base path used to resolve includes/imports within the provided XSD schema.                                             | None            |
+| `error_facets`  | `list[str]` | No        | The attributes of validation errors to collect and report (e.g., `path`, `reason`)                                     | [path, reason]  |
+| `fail_on_error` | `bool`      | No        | Whether to fail the test case if one or more XML validation errors are found. Can be overridden per keyword call.      | True            |
 
 ### Examples
 
@@ -151,7 +161,7 @@ Library    xmlvalidator    xsd_path=path/to/schema.xsd
 Library    xmlvalidator
 ```
 
-#### Importing with preloaded XSD that requires a base_url
+#### Importing with preloaded XSD that requires a `base_url`
 
 ```robotframework
 Library    xmlvalidator    xsd_path=path/to/schema_with_include.xsd
@@ -160,7 +170,7 @@ Library    xmlvalidator    xsd_path=path/to/schema_with_include.xsd
 
 Use `base_url` when your XSD uses `<xs:include>` or `<xs:import>` with relative paths.
 
-#### Importing with custom error_facets
+#### Importing with custom `error_facets`
 
 Use the `error_facets` argument to control which attributes of detected errors will be collected and reported.
 
@@ -179,6 +189,21 @@ Library    xmlvalidator    xsd_path=schemas/schema.xsd
 ...                        error_facets=value, namespaces
 ```
 
+#### Importing with `fail_on_errors=True
+
+The fail_on_errors argument controls whether a test case should fail if XML validation errors are detected.
+
+It defaults to True.
+
+The library's batch validation behavior remains unchanged. That is, `fail_on_errors=True` does *not* short-circuit the validation process in any way.
+
+Set `fail_on_errors=False` to log validation issues without failing the test. This is useful for:
+
+- Non-blocking checks in dashboards or QA reports.
+- Legacy or transitional systems where some invalid files are expected.
+- Schema discovery or diagnostics, where conformance isn’t yet enforced.
+- Soft rollout of stricter validation rules, allowing time to adapt.
+
 #### Further examples
 
 See also the [library initialization Robot test file](test/integration/01_library_initialization.robot).
@@ -189,39 +214,63 @@ See also the [library initialization Robot test file](test/integration/01_librar
 
 ### Keyword overview
 
-| Keyword                  | Description |
-|--------------------------|-------------|
+Thi section merely provides a short summary of the library's capabilities.
+
+For more details, please see the [keyword documentation](https://michaelhallik.github.io/robotframework-xmlvalidator/XmlValidator.html).
+
+| Keyword                  | Description                                                         |
+|--------------------------|---------------------------------------------------------------------|
 | `Validate Xml Files`     | Validate one or more XML files against one or more XSD schema files |
-| `Reset Schema`           | Clear the currently loaded XSD schema |
-| `Reset Errors`           | Clear the set of collected errors |
-| `Get Schema`             | Get the current schema name or object |
-| `Log Schema`             | Log the currently loaded schema |
-| `Get Error Facets`       | Returns a list of the currently active error facets |
-| `Reset Error Facets`     | Reset the error facets to default (`path`, `reason`) |
+| `Reset Schema`           | Clear the currently loaded XSD schema                               |
+| `Reset Errors`           | Clear the set of collected errors                                   |
+| `Get Schema`             | Get the current schema name or object                               |
+| `Log Schema`             | Log the currently loaded schema                                     |
+| `Get Error Facets`       | Returns a list of the currently active error facets                 |
+| `Reset Error Facets`     | Reset the error facets to default (`path`, `reason`)                |
 
 The main keyword is `Validate Xml Files`. The other keywords are convenience/helper functions, e.g. 'Reset Error Facets'.
 
+#### Error collection
+
 The `Validate Xml Files` validates one or more XML files against one or more XSD schema files and collects and reports all encountered errors.
 
-The type of error that the keyword can detect is not limited to XSD violations, but may also pertain to malformed XML files (e.g. parse errors), empty files, unmatched XML files (no XSD match found), etc.
+The type of error that the keyword can detect is not limited to XSD violations, but may also pertain to malformed XML files, empty files, unmatched XML files (no XSD match found), etc.
 
 Errors that result from malformed XML files or from XSD violations support detailed error reporting. Using the `error_facets` argument you may specify the details the keyword should collect and report about captured errors.
 
-When operating in batch mode, the `Validate Xml Files` keyword always validates the entire set of passed XML files.
+#### Batch mode
 
-That is, when it encounters an error in a file, it does not fail. Rather, it collects the error details (as determined by the `error_facets` arg) and then continues validating the current file as well as any subsequent file(s).
+The `Validate Xml Files` keyword always validates the entire set of passed XML files.
+
+That is, when it encounters an error in a file, it does not fail and stop execution. Rather, it collects the error details (as determined by the `error_facets` arg) and then continues validating the current file as well as any subsequent file(s).
 
 In that fashion the keyword works through the entire set of files.
 
-Once all files are processed it will log a summary of the test run and then proceed to report all collected errors in the console, in the RF log and, optionally, in the form of a CSV file.
+Once *all* files are processed it will log a summary of the test run and then proceed to report all collected errors in the console, in the RF log and, optionally, in the form of a CSV file.
 
-However, in case you want your test case to fail when one or more errors have been detected, you can use the ``fail_on_errors`` (bool) argument to make it so. It defaults to False. When setting it to True, the keyword will still check each XML file (and collect possible errors), but after it has thus processed the batch, it will fail if one or more errors will have been detected.
+For example:
+- If you validate fifteen XML files and five of them contain schema violations or other errors, all files will still be processed.
+- Errors are simply collected throughout the run and reported collectively, only after the final file has been (fully) processed.
+- The test case will fail (assuming `fail_on_errors=True`) only after all files have been checked, ensuring comprehensive diagnostics.
 
-The keyword further supports the dynamic matching (i.e. pairing) of XML and XSD files, using either a 'by filename' or a 'by namespace' strategy. That means you can simply pass the paths to a folder containing XML files and to a folder containing XSD files and the keyword will determine which XSD schema file to use for each XML file. If the XML and XSD files reside in the same folder, you only have to pass one folder path. When no matching XSD schema could be identified for an XML file, this will be integrated into the mentioned summary and error reporting (the keyword will not fail).
+#### Single file mode
 
 Of course, you may also refer to specific XML/XSD files (instead of to folders). In that case, no matching will be attempted, but the keyword will simply try to validate the specified XML file against the specified XSD file.
 
-For more details, please see the [keyword documentation](https://michaelhallik.github.io/robotframework-xmlvalidator/XmlValidator.html).
+Actually, almost anything goes:
+
+- one folder with: one or more XML files and one or more XSD files
+- one folder with one or more XML files and another folder with one or more XSD files
+- one folder with one or more XML files and a single XSD file
+- a single XML file and a sigle XSD file
+
+#### Test case status - fail_on_error
+
+A test case that has resulted in the collection of one or more errors (of whatever type) will receive a status of FAIL. You can use the ``fail_on_errors`` (bool) argument to change this default behaviour. When set to `False`, the test cases's will always be PASS, regardless whether errors were collected or not.
+
+#### Dyanmic XSD resolution
+
+The keyword further supports the dynamic matching (i.e. pairing) of XML and XSD files, using either a 'by filename' or a 'by namespace' strategy. That means you can simply pass the paths to a folder containing XML files and to a folder containing XSD files and the keyword will determine which XSD schema file to use for each XML file. If the XML and XSD files reside in the same folder, you only have to pass one folder path. When no matching XSD schema could be identified for an XML file, this will be integrated into the mentioned summary and error reporting (the keyword will not fail).
 
 ### Error collection
 
@@ -306,9 +355,9 @@ General errors that do not pertain to syntax or schema issues:
 
 #### Final note on error collection
 
-On account of the purpose of this library, all encountered errors (regardless the involved types) are collected and reported. The validator analyzes all files, collects encountered errors (if any) and, finally, reports the results of the run in the console and in the Robot Framework log 
+On account of the purpose of this library, all encountered errors (regardless the involved types) are collected and reported. The validator analyzes all files, collects encountered errors (if any) and, finally, reports the results of the run in the console and in the Robot Framework log.
 
-If you want the test run to receive a FAIL status when, at the end of test run, one or more errors have been found, then set: `fail_on_errors=True`.
+Every test case in which one or more errors have been collected, will receive status FAIL unless `fail_on_errors=True` (see earlier explanations).
 
 ### Keyword documentation
 
@@ -357,6 +406,10 @@ Validate Folder With Multiple Schemas By File Name
     Validate Xml Files    ${FOLDER_MULTIPLE_XML_XSD_FN}    xsd_search_strategy=by_file_name
 ```
 
+#### Demo test suite file as examples
+
+See the [demo test suite](test/demo/demo.robot) for a demo test suite that demonstrates the most important features of the library in a concise fashion.
+
 #### Integration tests as examples
 
 Note that the [integration test](test/integration) folder contains seven Robot Framework test suite files.
@@ -745,6 +798,18 @@ requirements.txt                     # Requirements file for users (pip)
 
 ---
 
+## Changelog
+
+For a list of changes across versions, including recent behavioral changes in validation logic, see the [CHANGELOG](https://github.com/MichaelHallik/robotframework-xmlvalidator/blob/main/CHANGELOG.md).
+
+---
+
+## Roadmap
+
+See the [project roadmap](ROADMAP.md) for upcoming features and ideas.
+
+---
+
 ## License
 
 Licensed under the Apache License 2.0. See [LICENSE](LICENSE).

ROADMAP.md

@@ -0,0 +1,46 @@
+# Project roadmap
+
+This document outlines planned features and/or ideas for the future development of the `robotframework-xmlvalidator` library.
+
+This roadmap is aspirational and may evolve based on user feedback input and practical constraints.
+
+---
+
+## In progress / near-term
+
+-
+
+---
+
+## Planned / future
+
+-
+
+---
+
+## Community wishlist / open for discussion
+
+-
+
+---
+
+## Ideas
+- Support for separate error facets for malformed XML errors vs. XSD violation errors.
+- XSD 1.1 support using XMLSchema 1.1 features (e.g., assertions, conditional types).
+- Explicit validation of XSD schemas using W3C meta-schema.
+- Flexible handling of wildcard elements (`<xs:any>`), with support for:
+  - `strict`: must be validatable
+  - `lax`: validate if possible
+  - `skip`: accept without validation
+- Support for additional schema types (e.g., RelaxNG, Schematron).
+- Optional CLI runner for validation outside of Robot Framework.
+- Export error details to more formats (e.g. JSON, XML).
+- Enbed error collection as HTML tables in the log file (inline validation summaries).
+- Built-in diffs between schema versions.
+- GUI frontend for non-technical users.
+
+---
+
+Last updated: 2025-06-13
+
+> Want to contribute or suggest a feature? Feel free to open an issue or join the discussion.

pyproject.toml

@@ -1,7 +1,7 @@
 # General.
 [tool.poetry]
 name = "robotframework-xmlvalidator"
-version = "1.0.1"
+version = "2.0.0a1"
 description = "A Robot Framework test library for validating XML files against XSD schemas"
 authors = ["Michael Hallik <hallik.michael@gmail.com>"]
 license = "Apache-2.0"