Skip to content

First version of tile based job splitter #756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

VincentVerelst
Copy link
Collaborator

@VincentVerelst VincentVerelst commented Apr 2, 2025

A first version of a split_area function based on a SizeBasedTileGrid class.

  • Accepts dict and shapely.geometry.Polygon objects
  • Resulting split tiles never exceed the bounds of the original AOI
  • Tile size can be specified in km or degree
  • For a more detailed explanation of how it behaves: see unit tests

closes #745

@soxofaan, @jdries, besides the general review, I could use your help to advice on how to expose this to users and leave flexibility of adding more TileGridInterface classes in the future.



# TODO: This function is also defined in openeo-python-driver. But maybe we want to avoid a dependency on openeo-python-driver?
def reproject_bounding_box(bbox: Dict, from_crs: Optional[str], to_crs: str) -> Dict:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we somehow mark this and actually all classes as private API?
By exposing minimal new API, we have to worry less when making changes, and can minimize the new API to review carefully.


def split_area(
aoi: Union[Dict, shapely.geometry.Polygon], projection="EPSG:3857", tile_size: float = 20.0
) -> List[Union[Dict, shapely.geometry.Polygon]]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Should we return only one type? Input can indeed be polygon, but output is always rectangular tiles? I would however return them as polygon as well.
  2. Many tiling systems also can assign a 'tile id', would be good if that is also returned

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so split_area should then return a geopandas.GeoDataFrame with columns tile_id and geometry?

Do you have any suggestions for the tile_id convention for a SizeBasedTileGrid?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's an example precomputed grid, which also has tile id's:
https://artifactory.vgt.vito.be/artifactory/auxdata-public/grids/LAEA-20km-EU27.geojson
I would propose that this splitter should be able to replicate the same grid with the same tile id's.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return a geopandas.GeoDataFrame with ...

note that geopandas is also not a core dependency of the client. It's only a test dependency

Copy link
Collaborator

@jdries jdries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logged some initial remarks on the API

Split area of interest into tiles of given size and projection.
:param aoi: area of interest (bounding box or shapely polygon)
:param projection: projection to use for splitting. Default is web mercator (EPSG:3857)
:param tile_size: size of tiles in km for UTM projections or degrees for WGS84
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:param tile_size: size of tiles in km for UTM projections or degrees for WGS84
:param tile_size: size of tiles in unit of measure of the projection

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me personally, this is confusing: the unit of measure of UTM projection is meter, but we expect tile_size to be in kilometer.

Copy link
Member

@soxofaan soxofaan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some quick notes

import math
from typing import Dict, List, NamedTuple, Optional, Union

import pyproj
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: pyproj is not a core dependency, only an optional one under the localprocessing extra at the moment.

Not sure if this dependency can be avoided, or is it a hard dependency for this whole module?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We indeed need to be able to reproject polygons in this module. Not sure if that's possible without pyproj? Possibly with geopandas, but that's also not a core dependency, and it built on pyproj I think

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can still do basic splitting without need for reprojection if you stay within the same CRS, right?
So I think it should be possible to keep pyproj an optional dependency

south: float
east: float
north: float
crs: str = "EPSG:4326"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: in strict openEO, EPSG CRS descriptors are integer codes, instead of the string format "EPSG:XXX". String CRS descriptors are assumed to be WKT2.

e.g. see https://github.com/Open-EO/openeo-processes/blob/6141771129d58d2a292db3b91d80c812c49f4e52/load_collection.json#L67

Coordinate reference system of the extent, specified as as EPSG code or WKT2 CRS string.

 "anyOf": [
                                {
                                    "title": "EPSG Code",
                                    "type": "integer",
                                    "subtype": "epsg-code",
                                    "minimum": 1000,
                                    "examples": [
                                        3857
                                    ]
                                },
                                {
                                    "title": "WKT2",
                                    "type": "string",
                                    "subtype": "wkt2-definition"
                                }
                            ],

If we introduce something new in the client I think we should follow that convention from the start.
I'm not sure actually why I did that wrong in the aggregator

return dict(zip(["west", "south", "east", "north"], reprojected.bounds), crs=to_crs)


# TODO: This class is also defined in openeo-aggregator. But maybe we want to avoid a dependency on openeo-aggregator?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we can not depend on aggregator from the client (the other way around is already happening, so that would actually be a circular dependency).
just remove this TODO

pass


# TODO: This function is also defined in openeo-python-driver. But maybe we want to avoid a dependency on openeo-python-driver?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we can not depend on openeo-python-driver from the client (the other way around is already happening, so that would actually be a circular dependency).
just remove this TODO


def split_area(
aoi: Union[Dict, shapely.geometry.Polygon], projection="EPSG:3857", tile_size: float = 20.0
) -> List[Union[Dict, shapely.geometry.Polygon]]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return a geopandas.GeoDataFrame with ...

note that geopandas is also not a core dependency of the client. It's only a test dependency

return cls(**{k: d[k] for k in cls._fields if k not in cls._field_defaults or k in d})

@classmethod
def from_polygon(cls, polygon: shapely.geometry.Polygon, projection: Optional[str] = None) -> "BoundingBox":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why only Polygon support and no MultiPolygon? I think we can support that here without any effort

Specification of a tile grid, parsed from a size and a projection.
"""

def __init__(self, epsg: str, size: float):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as mentioned above: epsg codes should be annotated as integers (that's just what epsg codes are).
at best I would normalize string "EPSG:123" to int 123 .

e.g. check out openeo.util.normalize_crs

"""Create a tile grid from size and projection"""
return cls(projection.lower(), size)

def get_tiles(self, geometry: Union[Dict, shapely.geometry.Polygon]) -> List[Union[Dict, shapely.geometry.Polygon]]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as mentioned above: I guess it's trivial to extend geometry support to MultiPolygons too

if isinstance(geometry, dict):
tiles.append(reproject_bounding_box(tile.as_dict(), from_crs=self.epsg, to_crs=bbox_crs))
else:
tiles.append(tile.as_polygon())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like risky behavior if I understand correctly: in one case you reproject and in another you don't. I'm not sure users will anticipate that when naively using this function.
I understand you can document that, but these kind of important details are easily glossed over.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's indeed assumed that the CRS of your Polygon is the same CRS as your SizeBasedTileGrid.

How would you suggest to detect the correct CRS of the Polygon?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at first sight I think this method tries to do too much (bboxes with crs, polygons without crs ..., reprojections, ...)

I'd extract the core functionality (double for-loop) as a helper and make its logic "CRS-free". And reuse that in more specialized versions (for CRS-aware bboxes, for naive polygons, ...)

Copy link
Collaborator Author

@VincentVerelst VincentVerelst Apr 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so in this first version we'll just not reproject anything to avoid dependency on pyproj and instead stay in the same crs, which can be specified via:

  • The crs key for a dict
  • The projection argument in split_area for Polygons and MultiPolygons

Or what do you propose?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I would make sure to have a core function that does just tile splitting in a naive way (in the sense that it does not considers any CRS aspects). This function would then not have to depend on pyproj.
Then, reuse this function for particular uses cases that possibly require reprojects, which is fine as long as it does not leak into the core splitting function.
Not sure if it's clear what I mean :)

"east": 100_000.0,
"north": 100_000.0,
"crs": "EPSG:3857",
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it's a matter of taste, but these four fixtures mock_polygon_wgs, ... mock_dict_with_crs_utm feel like overkill to me.
I guess it is to be DRY, but in testing I think this extra level of indirection for these simple constructs is worse for readability than violating DRY.
Also, if you still to be DRY, I think you should just use constants instead of pytest fixtures (you are not really using them as fixtures, in terms of setup/teardown)

x_offset = 0
else:
tile_size = self.size * 1000
x_offset = 500_000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this if-else looks too simple

  • epsg could be in upper case "EPSG:4326"
  • not being "epsg:4326" is not enough to be sure it's UTM

also: the multiplication with 1000 looks quite ad-hoc and based on undocumented assumptions. In UTM the base unit is meters, but here you seem to assume I guess that the tile size is given in km, but that is currently not clear

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Do you have any recommended utils on how to detect if a certain CRS is in UTM?

  • The expected units are indeed km for UTM (taken from the aggregator, can also just change it). It's documented in the split_area annotation, but I'll also add it to this class.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

epsg codes in ranges 32601-32660 and 32701-32760 are UTM crses.
e.g. also see https://github.com/Open-EO/openeo-python-driver/blob/af50be3f59f2152ff36c15512735343fe7e463c5/openeo_driver/util/utm.py#L29-L33

The base unit of UTM is meter, so using something else (even if documented) is asking for trouble somewhere down the line. If you want to support working with kilometer to make it user friendly, you should make that explicit in some way (e.g. in the aggregator, the tiling specification is expected to contain an explicit unit like "km" : https://github.com/Open-EO/openeo-aggregator/blob/a0d40582c813a4d1ed62221389fd5a776106c809/src/openeo_aggregator/partitionedjobs/splitting.py#L80-L103 )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then I propose we just assume tile_size to be in meter for UTM crs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

20km UTM tile splitter
3 participants