Skip to content

ETL V2 #69

@nicolas-kuechler

Description

@nicolas-kuechler

There are a few things that could be improved in the ETL results processing which may require a new version due to breaking changes.

These are things that should be considered:

  • Extractor performance is pretty bad
  • At the moment, ETL pipelines build a single data frame even when there are multiple result files involved.
  • ETL debugging could use more native support:
    etl-debug-design: install
        @cd $(does_config_dir) && \
        poetry run python -m debugpy --listen 5678 --wait-for-client $(PWD)/doespy/doespy/etl/etl.py --suite $(suite) --id $(id) --load_from_design
    

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions