Skip to content

StacApiWorkspace: use latest part of merge argument as collection id #1074

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
VictorVerhaert opened this issue Feb 27, 2025 · 3 comments · May be fixed by #1159
Open

StacApiWorkspace: use latest part of merge argument as collection id #1074

VictorVerhaert opened this issue Feb 27, 2025 · 3 comments · May be fixed by #1159
Assignees
@bossie
Labels
enhancement feature request STAC and load_stac

Comments

@VictorVerhaert
Copy link

the current implementation takes the merge argument of export_workspace as the collection id. This has however the implication that the merge argument can no longer contain slashes / as these are not allowed in collection id's.

This limitation disallows users to use the export_workspace process to export to different sub-folders of a workspace.

My proposed solution is to split the merge argument on / and take the last value as the collection id.

example:

configured workspace
  name: testworkspace
  bucket: mybucket
  stac api: my.stac.api
  prefix: mybucket/foo

export_workspace(
  workspace=testworkspace,
  merge= bar/colA
)

export_workspace(
  workspace=testworkspace,
  merge= zar/colB
)

bucket result:
mybucket
| foo
| | bar
| | | colA
| | zar
| | | colB

my.stac.api result:
colA
colB
@VictorVerhaert
Copy link
Author

As an alternative, and perhaps better, solution a new argument to export_workspace could be added with the collection_id. This would allow for the full freedom of the merge argument as we had before.
If this argument is left empty then we could fall back on the merge argument.
@jdries @soxofaan any feedback on this?

@soxofaan
Copy link
Member

soxofaan commented Mar 20, 2025 •
edited
Loading

I don't have a lot of practical export_workspace experience yet, but it seems reasonable to me to use the leaf of the merge path as collection id.

a new argument to export_workspace could be added with the collection_id

Note that if this solution would be picked, we have to move the discussion to https://github.com/Open-EO/openeo-processes to push it properly into the appropriate specs. This might be a lot more effort than just going for the convention to use the merge's leaf component as collection id.

@VictorVerhaert
Copy link
Author

I think we can do both
short term and as a default use the leaf of the merge argument
long term get the new argument in the openeo-processes approved

The reason we could need this long term solution is that projects often want to save their results grouped e.g. per year as subfolders within a collection

bossie added a commit that referenced this issue Apr 24, 2025
@bossie
support path as "merge" argument; last part becomes collection ID #1074
b640bbf
@bossie bossie linked a pull request Apr 24, 2025 that will close this issue
support path as "merge" argument; last part becomes collection ID #1159
Draft
@bossie bossie linked a pull request Apr 24, 2025 that will close this issue
Draft
bossie added a commit that referenced this issue Apr 24, 2025
@bossie
expose helper function #1074
086ed7a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bossie bossie

Labels
enhancement feature request STAC and load_stac
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

support path as "merge" argument; last part becomes collection ID Open-EO/openeo-geopyspark-driver
3 participants
@soxofaan @bossie @VictorVerhaert