Skip to content

20km UTM tile splitter #745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jdries opened this issue Feb 27, 2025 · 8 comments · May be fixed by #756
Open

20km UTM tile splitter #745

jdries opened this issue Feb 27, 2025 · 8 comments · May be fixed by #756
Assignees

Comments

@jdries
Copy link
Collaborator

jdries commented Feb 27, 2025

Example grid: https://artifactory.vgt.vito.be/ui/repos/tree/General/auxdata-public/grids/utm-tiling-grid-20km.gpkg

Define a function like this:

def split_area(aoi: Union[bounds, shapely.Geometry], projection="AUTO:42001", tile_size:float=20.0)

That simply returns a list of geometries that define grid cells, coordinates should be in projection.

Then we want to make it trivial to construct a job_db based on this, so wrap this code in a function:

df = gpd.GeoDataFrame({
    "geometry": split_area(aoi),
})
job_db = create_job_db("jobs.csv").initialize_from_df(df)
@jdries
Copy link
Collaborator Author

jdries commented Mar 18, 2025

@VincentVerelst see also this existing code for reuse:
https://github.com/Open-EO/openeo-aggregator/blob/master/src/openeo_aggregator/partitionedjobs/splitting.py#L113
To be checked with @soxofaan, but perhaps it's an option to simply move it to python client, where aggregator can then also find it.

@mbuchhorn
Copy link

that code is already existing in WEED jobmanager.... also have a look on my optimized global UTM 20km grid with the name grid20. That is reducing overburden in the processing and removed the flaws of the UTM and MGRS

@mbuchhorn
Copy link

my function construct hands back a standard openEO BBOX for the processing

@VincentVerelst
Copy link
Collaborator

VincentVerelst commented Mar 19, 2025

Currently, in GFMap we have these FeatureCollection splitters that group all geometries by S2 tile and return a list of all these groups. The splitting is based on this grid of non-overlapping S2 tiles, which I received from Kristof Van Tricht (can't tag him): https://artifactory.vgt.vito.be/artifactory/auxdata-public/gfmap/s2grid_voronoi_4326.parquet.

We could do the following:

  • Create a BoundingBox splitter to split a large BoundingBox into a list of smaller bounding boxes, based on the Voronoi grid above. This way we are consistent with the behaviour of the already existing FeatureCollection splitters
  • In a second step, add the option of setting a custom tilesize, based on the TileGrid class from the openeo aggregator
  • @mbuchhorn , I understand you already have code for efficient geometry intersection calculation, so we could use that as well

Things to discuss:

  • Which will be the default S2 grid we will use? Currently, it's this Voronoi grid sent over by Kristof, but I'm open to other suggestions, such as Marcel's grid. @mbuchhorn , Kristof Van Tricht, what do you think?
  • Where do we put these job splitters? Currently, we already have some splitters in GFMap, but I understand we want to include them in the Python Client? Which repo would you suggest @jdries , @soxofaan , @HansVRP ? I think it's best to have all splitters in one repo.

@jdries
Copy link
Collaborator Author

jdries commented Mar 19, 2025

@VincentVerelst this specific issue is not yet about feature splitting, it's about the easier case of splitting a single AOI into smaller ones, like in the example method signature I added in the description.
This should go into the python client repo, as a supported openEO python client feature.

@mbuchhorn
Copy link

@VincentVerelst @jdries I finish my "AOI_tiler" to be uniform for EU and globe context. If you can upload my grids also on the artifactory then it will be even more universal. The output of the AOI_splitter is always a (geo)pandas dataframe where you can then add some additional columns (like start, end, etc) and directly run in the jobmanager.

in the pandas dataframe you have a column called bbox_dict which is then the openEO BBOX representation of the tiled AOI. I use then to extra functions to convert it in the start_function back to a dict.

I had no time to allow as input to the "AOI_tiler" other as geopandas (in what ever crs you want). but I also plan to allow: BBOX (openEO BBOX format), shapely polygon (4326), GeoJson (only 4326) and THEN the output is always a (geo)pandas Dataframe with all data needed for the tiles.

i prepare some running examples for EU and global at the moment and then send it to you guys. use what you think is useful.
I hope to get it running between my meetings.

@mbuchhorn
Copy link

@jdries @VincentVerelst I wrote now the fully universal AOI splitter.... it eats as input every format you can think of (shapley polygon, openEO BBOX, Geopandas Dataframe, geoparquet or GeoJSON) and also the respected tiling grid can be everything from local file, loaded geoparquet from url, keyword for internal ressources or a GeoJSON object.

Output is a Geopandas dataframe with each row representing a tile (geometry) intersecting with the given AOI. You can optional also merge in columns from the original AOI file (like your parameters or so).

here is a Notebook which shows that all is working - I tested all possibilities of input against each other:
https://github.com/ESA-WEED-project/eo_processing/blob/mb_AOI_tiler/notebooks/AOI_tiling_for_jobmanager.ipynb

@mbuchhorn
Copy link

for feature extraction I do not need it since I follow there a different strategy, but even that should be not issue to implement in this function. function name is: AOI_tiler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants