-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Attaching a schema which describes necessary and produced columns of each task could aid in pre-execution validation. For example:
- Task A takes in "id" column, produces "data" column
- Task B takes in "id" and "data", produces "measurement"
- Task C takes "measurement", produces "result"
- Task D takes "measurement", "data", produces "another_result"
These could be executed in order [A, B, C, D] or [A, B, D, C], but not [A, D].
Since the aim is to validate a possible graph and enable plugging tasks in varying arrangements, we cannot define a full schema for tasks, only the necessary input columns and the columns produced, so existing schema definitions and validation tools might not fit.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request