Provide an ability to add custom validation rules?

[am17torres]

This library is great! It handles a lot of common validation constraints and coverage the vast majority of use cases. I'd really like to use this library but I have a few more complex business requirements for validations that are not covered natively. Before I put together a PR for this, I wanted to get some initial thoughts and feedback.

One such validation requirement is that the fields in a column need to exist in a remote system. I'm sure there are a number of other use cases other folks might have.

Here's what I am thinking:

In the field.constraints allow the user to pass in custom names and any arguments for that function.

{
  "fields": [
    {
        "name": "col",
        "constraints": {
            "required": true,
            "customFunction1": "scalarArgument", /* int, string, bool, etc. */
            "customFunction2": ["array", "of", 1, "mixed", true, "arguments"],
            "customFunction3": {
                "object": "of",
                "arguments": {
                    "including": "nested"
                }
            }
        }
    }
  ]
}

We can add new validation rules to the table. Perhaps something like table.customValidators() or similar. customValidators accepts a map of synchronous and async functions and should be defined before table.read or table.iter.

table.customValidators({
    customFunction1: async (row, ...args) => {
      // do some async validation
    },
    customFunction2: (row, ...args) => {
      // something sync
    }
    customFunction3:(row, ...args) => {
      // something sync
    }
})

This will allow the tableschema library more flexibility when it comes to validating data.

Thoughts?

[Stephen-Gates]

@am17torres sounds like may be looking for validating foreign keys across data packages.

See also https://github.com/frictionlessdata/goodtables-py#advanced-checks

[roll]

@am17torres
That's interesting!

You could also take a look at goodtables family:

• https://www.goodtables.io
• https://github.com/frictionlessdata/goodtables-js
• https://github.com/frictionlessdata/goodtables-py

goodtables is designed to be used for very complex business validation rules. Unfortunately, for now, there is only a wrapper for JavaScript based on goodtables.io API.

PS.
But goodtables is exactly a framework for custom validation rules 😃

[am17torres]

Thanks for the idea using goodtables I was not aware it existed before now. Unfortunately, that won't work for my use case as the microservice (running node/express) responsible for validating the uploaded CSV will not have public internet access so goodtables.io won't be reachable.

I suppose I could, in theory, create a lambda or similar running python which could be reachable from within the private network, but that seems like a lot of extra effort and infrastructure.