Skip to content

Check if zarr store supports consolidated metadata #10457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
01e7518
new blank whatsnew
TomNicholas Oct 24, 2024
83e553b
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Oct 24, 2024
e44326d
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 8, 2024
4e4eeb0
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 20, 2024
d858059
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 21, 2024
d377780
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 21, 2024
3132f6a
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 23, 2024
900eef5
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Nov 29, 2024
4c4462f
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Dec 4, 2024
5b9b749
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jan 6, 2025
fadb953
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jan 8, 2025
57d9d23
Merge branch 'main' of https://github.com/TomNicholas/xarray
TomNicholas Mar 13, 2025
11170fc
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 19, 2025
0b8fa41
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 20, 2025
f769f85
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Mar 20, 2025
4eef318
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Apr 7, 2025
29242a4
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas May 15, 2025
5d15bbd
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas May 19, 2025
b4ed8ee
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jun 20, 2025
6a23bfb
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jun 24, 2025
d87b209
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jun 27, 2025
08cb041
add test for store which explicitly does not support consolidated met…
TomNicholas Jun 27, 2025
9cc9105
check for consolidated metadata support
TomNicholas Jun 27, 2025
c817869
WrapperStore now only imported and used if zarr v3
TomNicholas Jun 28, 2025
50cff3e
use dedicated decorator
TomNicholas Jun 28, 2025
a47b2e2
use getattr in case property not defined
TomNicholas Jun 28, 2025
76f16fe
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 28, 2025
9b39529
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jun 28, 2025
e174d9c
implement with_read_only
TomNicholas Jun 30, 2025
5b441b2
Merge branch 'check_supports_consolidated_metadata' of https://github…
TomNicholas Jun 30, 2025
e395c60
Merge branch 'main' into check_supports_consolidated_metadata
TomNicholas Jul 1, 2025
1829807
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jul 1, 2025
de02ced
Merge branch 'main' of https://github.com/pydata/xarray
TomNicholas Jul 2, 2025
8167b83
Merge branch 'main' into check_supports_consolidated_metadata
TomNicholas Jul 2, 2025
1e30664
Merge branch 'check_supports_consolidated_metadata' of https://github…
TomNicholas Jul 2, 2025
101b280
change test to not expect warning if consolidated metadata not supported
TomNicholas Jul 2, 2025
26a402c
move store definition outside of test
TomNicholas Jul 2, 2025
1ce9d87
try setting Store=None when zarr-python v3 not installed
TomNicholas Jul 2, 2025
ad4d79c
remove Store type hint entirely to avoid import issues
TomNicholas Jul 2, 2025
09915dd
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 2, 2025
8252f9f
try WrapperStore = object
TomNicholas Jul 2, 2025
e550851
ensure WrapperStore is defined as something even if zarr isn't presen…
TomNicholas Jul 2, 2025
470e701
Merge branch 'main' into check_supports_consolidated_metadata
TomNicholas Jul 2, 2025
586c7a2
release note
TomNicholas Jul 2, 2025
e673608
Merge branch 'check_supports_consolidated_metadata' of https://github…
TomNicholas Jul 2, 2025
758eea4
correct RST syntax
TomNicholas Jul 2, 2025
e3f0d7b
Merge branch 'main' into check_supports_consolidated_metadata
TomNicholas Jul 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ New Features
~~~~~~~~~~~~
- Expose :py:class:`~xarray.indexes.RangeIndex`, and :py:class:`~xarray.indexes.CoordinateTransformIndex` as public api
under the ``xarray.indexes`` namespace. By `Deepak Cherian <https://github.com/dcherian>`_.
- Support zarr-python's new ``.supports_consolidated_metadata`` store property (:pull:`10457``).
by Tom Nicholas <https://github.com/TomNicholas>`_.
- Better error messages when encoding data to be written to disk fails (:pull:`10464`).
By `Stephan Hoyer <https://github.com/shoyer>`_

Expand Down
6 changes: 6 additions & 0 deletions xarray/backends/zarr.py
Original file line number Diff line number Diff line change
Expand Up @@ -1768,6 +1768,11 @@ def _get_open_params(
else:
missing_exc = zarr.errors.GroupNotFoundError

if _zarr_v3():
# zarr 3.0.8 and earlier did not support this property - it was effectively assumed true
if not getattr(store, "supports_consolidated_metadata", True):
consolidated = consolidate_on_close = False

if consolidated in [None, True]:
# open the root of the store, in case there is metadata consolidated there
group = open_kwargs.pop("path")
Expand Down Expand Up @@ -1825,6 +1830,7 @@ def _get_open_params(
else:
# this was the default for v2 and should apply to most existing Zarr data
use_zarr_fill_value_as_mask = True

return (
zarr_group,
consolidate_on_close,
Expand Down
54 changes: 48 additions & 6 deletions xarray/tests/test_backends.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@
requires_scipy,
requires_scipy_or_netCDF4,
requires_zarr,
requires_zarr_v3,
)
from xarray.tests.test_coding_times import (
_ALL_CALENDARS,
Expand Down Expand Up @@ -117,6 +118,7 @@

if has_zarr_v3:
from zarr.storage import MemoryStore as KVStore
from zarr.storage import WrapperStore

ZARR_FORMATS = [2, 3]
else:
Expand All @@ -127,8 +129,11 @@
)
except ImportError:
KVStore = None # type: ignore[assignment,misc,unused-ignore]

WrapperStore = object # type: ignore[assignment,misc,unused-ignore]
else:
KVStore = None # type: ignore[assignment,misc,unused-ignore]
WrapperStore = object # type: ignore[assignment,misc,unused-ignore]
ZARR_FORMATS = []


Expand Down Expand Up @@ -2400,12 +2405,13 @@ def test_read_non_consolidated_warning(self) -> None:
self.save(
expected, store_target=store, consolidated=False, **self.version_kwargs
)
with pytest.warns(
RuntimeWarning,
match="Failed to open Zarr store with consolidated",
):
with xr.open_zarr(store, **self.version_kwargs) as ds:
assert_identical(ds, expected)
if getattr(store, "supports_consolidated_metadata", True):
with pytest.warns(
RuntimeWarning,
match="Failed to open Zarr store with consolidated",
):
with xr.open_zarr(store, **self.version_kwargs) as ds:
assert_identical(ds, expected)

def test_non_existent_store(self) -> None:
with pytest.raises(
Expand Down Expand Up @@ -3756,6 +3762,42 @@ def test_chunk_key_encoding_v2(self) -> None:
assert actual["var1"].encoding["chunks"] == (2, 2)


class NoConsolidatedMetadataSupportStore(WrapperStore):
"""
Store that explicitly does not support consolidated metadata.

Useful as a proxy for stores like Icechunk, see https://github.com/zarr-developers/zarr-python/pull/3119.
"""

supports_consolidated_metadata = False

def __init__(
self,
store,
*,
read_only: bool = False,
) -> None:
self._store = store.with_read_only(read_only=read_only)

def with_read_only(
self, read_only: bool = False
) -> NoConsolidatedMetadataSupportStore:
return type(self)(
store=self._store,
read_only=read_only,
)


@requires_zarr_v3
class TestZarrNoConsolidatedMetadataSupport(ZarrBase):
@contextlib.contextmanager
def create_zarr_target(self):
# TODO the zarr version would need to be >3.08 for the supports_consolidated_metadata property to have any effect
yield NoConsolidatedMetadataSupportStore(
zarr.storage.MemoryStore({}, read_only=False)
)


@requires_zarr
@pytest.mark.skipif(
ON_WINDOWS,
Expand Down
Loading