Check if zarr store supports consolidated metadata #10457

TomNicholas · 2025-06-27T23:10:59Z

Needed to support the addition of .supports_consolidated_metadata in zarr-python upstream: zarr-developers/zarr-python#3119.

Closes #xxxx
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
~~New functions/methods are listed in api.rst~~

…adata

TomNicholas · 2025-06-27T23:12:07Z

xarray/tests/test_backends.py

@@ -3744,6 +3746,29 @@ def test_chunk_key_encoding_v2(self) -> None:
                assert actual["var1"].encoding["chunks"] == (2, 2)


+class NoConsolidatedMetadataSupportStore(WrapperStore[Store]):


TODO: I don't think I even want to try and define this class if zarr-python v3 is not installed

jhamman · 2025-06-27T23:13:35Z

xarray/backends/zarr.py

@@ -1768,6 +1768,10 @@ def _get_open_params(
    else:
        missing_exc = zarr.errors.GroupNotFoundError

+    if _zarr_v3():
+        if not store.supports_consolidated_metadata:


Suggested change

if not store.supports_consolidated_metadata:

if not getattr(store, "supports_consolidated_metadata", True):

I think this will be needed for Zarr >3.0.0,<3.1.0

Right - I had something like that then removed it 🤦

added in a47b2e2

paraseba · 2025-06-30T18:43:01Z

Thank you @maxrjones . Long term, I'd love to see the responsibility for managing the readonly flag removed from the abstract Store hierarchy. It feels this is a concern that is beyond what a Store should know about. It would help us keeping the Stores abstract and focused. Readonly-ness sounds like a session concern, independent of the details of how to write zarr objects to different media.

TomNicholas · 2025-06-30T18:59:23Z

Thanks for the discussion everyone - I want to move any further discussion of this over to zarr-developers/zarr-python#3186, where I've raised @paraseba's suggestion as an issue.

For now I think I have what I need to be able to finish this PR.

TomNicholas · 2025-07-01T19:21:52Z

So for this PR obviously there are failures here, but:

Upstream includes unreleased zarr data types changes, which will break a lot of things until @ianhi 's PR Updates for Zarr 3 Dtypes #10456 goes in - these are not needed for compatibility with zarr 3.0.9
Most of the failures are easy to fix (things like regexes getting differently-worded error messages than they expected) EDIT: Apparently @dcherian already fixed this one in main in Update error message regex for latest Zarr #10485, tracked by remove xfail on test_to_zarr_zip_store in test_backends_datatree #10486
I think the AttributeError: Can't get local object 'TestZarrNoConsolidatedMetadataSupport.create_zarr_target.<locals>.NoConsolidatedMetadataSupportStore' were caused by me hiding the definition of my test store in the test itself, and don't indicate a real bug EDIT: Defining it inside caused it to be unpickleable, which breaks some of the tests
There are still Store is not read-only but mode is 'r', for tests where we use ZipStore, because IIUC that does not implement .with_read_only. As this is a breaking change the only way to work around it would be to change xarray's tests to use a read-only zip store, and that should technically be done in a separate PR to this one. EDIT: See How to roundtrip to a ZipStore? zarr-developers/zarr-python#3194 and the temporary workaround remove xfail on test_to_zarr_zip_store in test_backends_datatree #10486
There is a failure where reading a non-consolidated store does not warn when it's expected to warn - I think that does indicate a logic bug in this PR for me to fix EDIT: addressed in 101b280, the test was expecting a warning that doesn't makes sense to be thrown for a store which does not support consolidated metadata

But that's it - then xarray should work with zarr 3.0.9 (and icechunk).

….com/TomNicholas/xarray into check_supports_consolidated_metadata

for more information, see https://pre-commit.ci

…t at all

TomNicholas · 2025-07-02T18:54:42Z

There is also an significant upstream performance regression caught by tests here that I am trying to ignore in Downgrade zarr requests performance expectations for zarr 3.0.9 #10491

….com/TomNicholas/xarray into check_supports_consolidated_metadata

TomNicholas and others added 23 commits October 24, 2024 17:48

new blank whatsnew

01e7518

Merge branch 'main' of https://github.com/pydata/xarray

83e553b

Merge branch 'main' of https://github.com/pydata/xarray

e44326d

Merge branch 'main' of https://github.com/pydata/xarray

4e4eeb0

Merge branch 'main' of https://github.com/pydata/xarray

d858059

Merge branch 'main' of https://github.com/pydata/xarray

d377780

Merge branch 'main' of https://github.com/pydata/xarray

3132f6a

Merge branch 'main' of https://github.com/pydata/xarray

900eef5

Merge branch 'main' of https://github.com/pydata/xarray

4c4462f

Merge branch 'main' of https://github.com/pydata/xarray

5b9b749

Merge branch 'main' of https://github.com/pydata/xarray

fadb953

Merge branch 'main' of https://github.com/TomNicholas/xarray

57d9d23

Merge branch 'main' of https://github.com/pydata/xarray

11170fc

Merge branch 'main' of https://github.com/pydata/xarray

0b8fa41

Merge branch 'main' of https://github.com/pydata/xarray

f769f85

Merge branch 'main' of https://github.com/pydata/xarray

4eef318

Merge branch 'main' of https://github.com/pydata/xarray

29242a4

Merge branch 'main' of https://github.com/pydata/xarray

5d15bbd

Merge branch 'main' of https://github.com/pydata/xarray

b4ed8ee

Merge branch 'main' of https://github.com/pydata/xarray

6a23bfb

Merge branch 'main' of https://github.com/pydata/xarray

d87b209

add test for store which explicitly does not support consolidated met…

08cb041

…adata

check for consolidated metadata support

9cc9105

TomNicholas added the topic-zarr Related to zarr storage library label Jun 27, 2025

github-actions bot added topic-backends topic-documentation io labels Jun 27, 2025

TomNicholas commented Jun 27, 2025

View reviewed changes

jhamman reviewed Jun 27, 2025

View reviewed changes

WrapperStore now only imported and used if zarr v3

c817869

TomNicholas mentioned this pull request Jun 30, 2025

Remove responsibility for managing readonly flag from abstract Store hierarchy? zarr-developers/zarr-python#3186

Open

Merge branch 'main' into check_supports_consolidated_metadata

e395c60

TomNicholas added the run-upstream Run upstream CI label Jul 1, 2025

Merge branch 'main' of https://github.com/pydata/xarray

1829807

TomNicholas mentioned this pull request Jul 2, 2025

How to roundtrip to a ZipStore? zarr-developers/zarr-python#3194

Open

TomNicholas and others added 10 commits July 2, 2025 10:34

Merge branch 'main' of https://github.com/pydata/xarray

de02ced

Merge branch 'main' into check_supports_consolidated_metadata

8167b83

Merge branch 'check_supports_consolidated_metadata' of https://github…

1e30664

….com/TomNicholas/xarray into check_supports_consolidated_metadata

change test to not expect warning if consolidated metadata not supported

101b280

move store definition outside of test

26a402c

try setting Store=None when zarr-python v3 not installed

1ce9d87

remove Store type hint entirely to avoid import issues

ad4d79c

[pre-commit.ci] auto fixes from pre-commit.com hooks

09915dd

for more information, see https://pre-commit.ci

try WrapperStore = object

8252f9f

ensure WrapperStore is defined as something even if zarr isn't presen…

e550851

…t at all

TomNicholas added 4 commits July 2, 2025 12:21

Merge branch 'main' into check_supports_consolidated_metadata

470e701

release note

586c7a2

Merge branch 'check_supports_consolidated_metadata' of https://github…

e673608

….com/TomNicholas/xarray into check_supports_consolidated_metadata

correct RST syntax

758eea4

rabernat approved these changes Jul 2, 2025

View reviewed changes

paraseba approved these changes Jul 2, 2025

View reviewed changes

Merge branch 'main' into check_supports_consolidated_metadata

e3f0d7b

TomNicholas enabled auto-merge (squash) July 2, 2025 19:53

TomNicholas disabled auto-merge July 2, 2025 19:53

TomNicholas merged commit ad25833 into pydata:main Jul 2, 2025
29 of 32 checks passed

TomNicholas deleted the check_supports_consolidated_metadata branch July 2, 2025 19:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Check if zarr store supports consolidated metadata #10457

Check if zarr store supports consolidated metadata #10457

Uh oh!

TomNicholas commented Jun 27, 2025 •

edited

Loading

Uh oh!

TomNicholas Jun 27, 2025 •

edited

Loading

Uh oh!

jhamman Jun 27, 2025

Uh oh!

TomNicholas Jun 27, 2025

Uh oh!

TomNicholas Jun 28, 2025

Uh oh!

paraseba commented Jun 30, 2025

Uh oh!

TomNicholas commented Jun 30, 2025 •

edited

Loading

Uh oh!

TomNicholas commented Jul 1, 2025 •

edited

Loading

Uh oh!

TomNicholas commented Jul 2, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

		@@ -3744,6 +3746,29 @@ def test_chunk_key_encoding_v2(self) -> None:
		assert actual["var1"].encoding["chunks"] == (2, 2)


		class NoConsolidatedMetadataSupportStore(WrapperStore[Store]):

	if not store.supports_consolidated_metadata:
	if not getattr(store, "supports_consolidated_metadata", True):

Uh oh!

Check if zarr store supports consolidated metadata #10457

Check if zarr store supports consolidated metadata #10457

Uh oh!

Conversation

TomNicholas commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomNicholas Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhamman Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

TomNicholas Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

TomNicholas Jun 28, 2025

Choose a reason for hiding this comment

Uh oh!

paraseba commented Jun 30, 2025

Uh oh!

TomNicholas commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomNicholas commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomNicholas commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomNicholas commented Jun 27, 2025 •

edited

Loading

TomNicholas Jun 27, 2025 •

edited

Loading

TomNicholas commented Jun 30, 2025 •

edited

Loading

TomNicholas commented Jul 1, 2025 •

edited

Loading

TomNicholas commented Jul 2, 2025 •

edited

Loading