Skip to content

Add configurability to custom Steps #915

@neolefty

Description

@neolefty

Overview

A research team within the Purdue College of Agriculture has started to use Frictionless for its data pipelines, and we would like to write custom Steps that receive configuration options from the Pipeline YAML file. For example:

steps:

  # Standard Step -- ensure ID is an integer
  - type: field-update
    name: id
    descriptor:
      name: farm_animal_id
      type: integer

  # Custom Step -- normalize funky date formats
  # Currently, in our testing, values (below) such as field_name or descriptor.name
  # are not getting passed to the custom step, and it appears they are not kept by
  # the custom Step mechanism. We want to capture them and pass them into the
  # custom Step initialization.
  - type: date-normalize
    field_name: Date
    descriptor:
      name: Date
      type: string

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions