-
Notifications
You must be signed in to change notification settings - Fork 309
Refactor resampler module to submodules #3124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
The other thing about this PR that I'm realizing is that some of the resamplers (ewa maybe? kdtree nearest maybe?) exist in pyresample in a way that should be useable. Or at least that was my intention before I stopped working on pyresample 2.0. Maybe it never got all the way there (as we already know I'm wrong about the caching functionality implemented in pyresample). |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3124 +/- ##
==========================================
- Coverage 96.24% 96.23% -0.01%
==========================================
Files 421 429 +8
Lines 57671 57777 +106
==========================================
+ Hits 55506 55603 +97
- Misses 2165 2174 +9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Pull Request Test Coverage Report for Build 15308592336Details
💛 - Coveralls |
I think this is now ready. Unless we come up with a better name for the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One more comment for making things private, but also what do you think about moving get_area_file/def
functions and add_xy_coords
and add_crs_xy_coords
to a utils.py
module? Or even go as far as the get_area_*
functions to one module outside of the resample subpackage (since they are used by readers and users) like satpy.area
or satpy.area_utils
or satpy.area_config
? And add_*_coords
to another module inside or outside of resampling? Like satpy.coords
or satpy.coords_utils
?
Oh another thought: if we imagine that the base resampler class(es) will all end up moving to pyresample and possibly even the "prepare_resampler" or "get_resampler"-style functions would be moved. Does that change the naming? Or rather, maybe those types of functions should not be in |
I'll have a look at this today evening after I reach the hotel in Milan, where I need to stay overnight due to a cancelled flight. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome. I noticed one small thing that could be changed and cleaned up. The only other discussion we should maybe have is how many satpy/X_utils.py
modules do we have before making things a series of utils/X.py
modules? Or would it be better to drop the _utils.py
part of the name and just call them satpy/area.py
and satpy/coords.py
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! Just. Awesome.
I think the most obvious win from this PR is how many things are loading from the area and coords utils instead of the resample module/subpackage.
I made a couple inline comments about long term design decisions. The only other remaining question for me is should satpy/area_utils.py
and coords_utils.py
have the _utils
removed?
if paths: | ||
return paths | ||
else: | ||
return get_config_path("areas.def") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In a near-future PR should we drop support for the old areas format?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think it's well due time. At the latest in 1.0.
try: | ||
from .native import get_native_resampler_classes | ||
resamplers.update(get_native_resampler_classes()) | ||
except ImportError: | ||
pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it needs to be done in this PR but this seems like another case where it would be easy to have a dictionary or list of strings to import and use importlib.import_module
and getattr
to get access to them. You could also use with suppress(ImportError):
to avoid the "uglier" except ImportError
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually considered with suppress():
for a sec yesterday, but got lazy. If I don't have anything better to do tomorrow, I'll have another look at both options.
def get_ewa_resampler_classes(): | ||
"""Get bucket resampler classes.""" | ||
return { | ||
"ewa": DaskEWAResampler, | ||
"ewa_legacy": LegacyDaskEWAResampler, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the idea that this entire module would go away if/when Resamplers can be loaded directly from a pyresample interface? Or should something like this be replace with a dict/list of strings like I mentioned in the base.py
comment? That is a series of strings to be imported.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea was to just kind of follow the structure of the other resample modules and provide a single function to get a dict of the available resamplers to replace the RESAMPLERS
dict in __init__.py
but still have the ability to get all of them.
from satpy.composites import IncompatibleAreas | ||
from satpy.composites.config_loader import load_compositor_configs_for_sensors | ||
from satpy.dataset import DataID, DataQuery, DatasetDict, combine_metadata, dataset_walker, replace_anc | ||
from satpy.dependency_tree import DependencyTree | ||
from satpy.node import CompositorNode, MissingDependencies, ReaderNode | ||
from satpy.readers.core.loading import load_readers | ||
from satpy.resample import get_area_def, prepare_resampler, resample_dataset | ||
from satpy.resample.base import prepare_resampler, resample_dataset |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since pyresample is a hard requirement this probably doesn't matter, but I'm wondering if this import should be in-function to avoid import Resampler classes. 🤷♂️ Eh that will probably go away naturally when more things are moved to pyresample.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, certainly. I'll move it in-function tomorrow if I don't have anything else. Would help in the short-term anyways.
This PR refactors the
satpy/resample.py
module to asatpy.resample
subpackage and splits the contents to several modules.At least the warnings in the tests should be fixed, but this is the first version to see if there are other things to consider. Also some of the imports could be moved to where they are used. And the docstring should be fixed with the new import paths.
Triggered by #3123
AUTHORS.md
if not there already