[Request] Allow uploading of additional files to artifact storage

[danricbede]

This request mainly comes from issues with integration tests, where we'd like to reference files from other project directories.

As an example, assume we have a directory structure like so:

+ src
+ tests
+ json
- pyproject.toml

When we define our Workflow/Task in the deployment.yml file we're using pytest as follows:

            spark_python_task:
              python_file: "file://tests/entrypoint.py"
              parameters: [ "file:fuse://tests/integration", "-k mytest" ]

Problem being when we run dbx execute, it only uploads the wheel and any files referenced in the spark_python_task section.

To get around this, a hack that I've used is to do the following:

            spark_python_task:
              python_file: "file://tests/entrypoint.py"
              parameters: [ "file:fuse://tests/integration", "-k mytest", " file:fuse://json"]

This isn't ideal for many reasons.

Request: New class attribute in spark_python_task that allows us to specificy a list of additional directories/files to upload.