Merge branch 'main' into to-iris

Author: charles-turner-1

.github/ISSUE_TEMPLATE/bug_report.md

@@ -17,7 +17,6 @@ Here's a quick checklist in what to include:
 - [ ] Include a detailed description of the bug or suggestion
 - [ ] Output of `intake_esm.show_versions()`
 - [ ] Minimal, self-contained copy-pastable example that generates the issue if possible. Please be concise with code posted. See guidelines below on how to provide a good bug report:
-
   - [Minimal Complete Verifiable Examples](https://stackoverflow.com/help/mcve)
   - [Craft Minimal Bug Reports](http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports)
 

.github/workflows/ci.yaml

@@ -52,7 +52,7 @@ jobs:
           python -m pytest
 
       - name: Upload code coverage to Codecov
-        uses: codecov/codecov-action@v5.4.2
+        uses: codecov/codecov-action@v5.4.3
         with:
           file: ./coverage.xml
           flags: unittests

.pre-commit-config.yaml

@@ -15,7 +15,7 @@ repos:
       - id: mixed-line-ending
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.11.8"
+    rev: "v0.12.7"
     hooks:
       - id: ruff
         args: ["--fix"]

CHANGELOG.md

@@ -1,5 +1,80 @@
 # Changelog
 
+[Full Changelog](https://github.com/intake/intake-esm/compare/v2025.2.3...v2025.7.9)
+
+## v2025.7.9
+
+### New features added
+
+- Support Python313, add setuptools to requirements by @Zeitsperre in https://github.com/intake/intake-esm/pull/707
+- Load with polars by @charles-turner-1 in https://github.com/intake/intake-esm/pull/709
+- Add interactive view of catalog by @charles-turner-1 in https://github.com/intake/intake-esm/pull/723
+
+### Bugs fixed
+
+- Fixed bug where pyarrow conversions were causing string accessor to fail in search by @charles-turner-1 in https://github.com/intake/intake-esm/pull/718
+
+### Maintenance and upkeep improvements
+
+- Update default argument for decode_timedelta by @charles-turner-1 in https://github.com/intake/intake-esm/pull/706
+- 697- Fix segfault by @charles-turner-1 in https://github.com/intake/intake-esm/pull/712
+- Fix broken `require_all_on` example in docs by @charles-turner-1 in https://github.com/intake/intake-esm/pull/720
+- Improve test performance by @charles-turner-1 in https://github.com/intake/intake-esm/pull/719
+
+### Other merged PRs
+
+- Bump codecov/codecov-action from 5.3.1 to 5.4.0 in the actions group by @dependabot in https://github.com/intake/intake-esm/pull/703
+- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/intake/intake-esm/pull/704
+- Remove setuptools runtime requirements by @Zeitsperre in https://github.com/intake/intake-esm/pull/708
+- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/intake/intake-esm/pull/713
+- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/intake/intake-esm/pull/716
+- Bump codecov/codecov-action from 5.4.0 to 5.4.2 in the actions group by @dependabot in https://github.com/intake/intake-esm/pull/714
+- Bump codecov/codecov-action from 5.4.2 to 5.4.3 in the actions group by @dependabot in https://github.com/intake/intake-esm/pull/721
+- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/intake/intake-esm/pull/722
+- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/intake/intake-esm/pull/726
+
+### New Contributors
+
+- @Zeitsperre made their first contribution in https://github.com/intake/intake-esm/pull/707
+
+## v2025.2.3
+
+[Full Changelog](https://github.com/intake/intake-esm/compare/v2024.2.6...v2025.2.3)
+
+### New features added
+
+- Changed behaviour of `source._open_dataset` to include coordinate variables by @charles-turner-1 in https://github.com/intake/intake-esm/pull/681
+- feat: Support being able to use in-memory ESMCatalogModel instances by @lewisjared in https://github.com/intake/intake-esm/pull/690
+
+### Bugs fixed
+
+- Fix #684 Merge results from main and derived catalogs by @rbeucher in https://github.com/intake/intake-esm/pull/685
+- Fix how storage_options is passed to get_mapper by @garciampred in https://github.com/intake/intake-esm/pull/678
+
+### Maintenance and upkeep improvements
+
+- Upgrade to intake Take2 by @charles-turner-1 in https://github.com/intake/intake-esm/pull/683
+- Update dependencies and improve import handling for xarray version compatibility by @andersy005 in https://github.com/intake/intake-esm/pull/696
+- Update pyproject.toml and readthedocs.yml by @andersy005 in https://github.com/intake/intake-esm/pull/695
+- Add flaky decorator for tests with remote resources to reduce cold start failure rate by @charles-turner-1 in https://github.com/intake/intake-esm/pull/700
+- Update CONTRIBUTING guidelines to e.g. refer to new default branch by @sadielbartholomew in https://github.com/intake/intake-esm/pull/674
+
+### Other merged PRs
+
+- Bump the actions group with 16 updates by @dependabot in https://github.com/intake/intake-esm/pull/658, https://github.com/intake/intake-esm/pull/654, https://github.com/intake/intake-esm/pull/661, https://github.com/intake/intake-esm/pull/671, https://github.com/intake/intake-esm/pull/686, https://github.com/intake/intake-esm/pull/691, https://github.com/intake/intake-esm/pull/693, https://github.com/intake/intake-esm/pull/698
+- Update PyPI workflow to build and upload intake-esm artifacts by @andersy005 in https://github.com/intake/intake-esm/pull/659
+- [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/intake/intake-esm/pull/662, https://github.com/intake/intake-esm/pull/664, https://github.com/intake/intake-esm/pull/669, https://github.com/intake/intake-esm/pull/672, https://github.com/intake/intake-esm/pull/675, https://github.com/intake/intake-esm/pull/677, https://github.com/intake/intake-esm/pull/682, https://github.com/intake/intake-esm/pull/687, https://github.com/intake/intake-esm/pull/692, https://github.com/intake/intake-esm/pull/694
+- Bump codecov/codecov-action from 4.1.1 to 4.4.1 in the actions group by @dependabot in https://github.com/intake/intake-esm/pull/663,https://github.com/intake/intake-esm/pull/668
+- Bump pypa/gh-action-pypi-publish from 1.9.0 to 1.10.2 in the actions group by @dependabot in https://github.com/intake/intake-esm/pull/676, https://github.com/intake/intake-esm/pull/680
+
+### New Contributors
+
+- @sadielbartholomew made their first contribution in https://github.com/intake/intake-esm/pull/674
+- @garciampred made their first contribution in https://github.com/intake/intake-esm/pull/678
+- @charles-turner-1 made their first contribution in https://github.com/intake/intake-esm/pull/681
+- @rbeucher made their first contribution in https://github.com/intake/intake-esm/pull/685
+- @lewisjared made their first contribution in https://github.com/intake/intake-esm/pull/690
+
 ## v2024.2.6
 
 ([full changelog](https://github.com/intake/intake-esm/compare/v2023.10.27...d96efea14b348e346d5c2a1490e91c9ae1e2c709))
@@ -684,7 +759,6 @@
 
 - Fix CESM-LE ice component peculiarities that caused intake-esm to load data improperly.
   The fix separates variables for `ice` component into two separate components:
-
   - `ice_sh`: for southern hemisphere
   - `ice_nh`: for northern hemisphere
 

README.md

@@ -64,7 +64,7 @@ providing necessary functionality for searching, discovering, data access/loadin
      ...: )
 
   In [6]: cat_subset
-  Out[6]: <GOOGLE-CMIP6 catalog with 4 dataset(s) from 261 asset(s)>
+  Out[6]: <GOOGLE-CMIP6 catalog with 2 dataset(s) from 67 asset(s)>
   ```
 
 - Access: when the user is satisfied with the results of their query, they can load data assets (netCDF and/or Zarr stores) into xarray datasets:

ci/environment-docs.yml

@@ -10,6 +10,7 @@ dependencies:
   - fsspec >=2024.12
   - gcsfs >=2024.12
   - intake >=2.0
+  - itables
   - jupyterlab
   - matplotlib
   - myst-nb
@@ -29,7 +30,7 @@ dependencies:
   - watermark
   - xarray-datatree >=0.0.9
   - xarray >=2024.10
-  - zarr >=2.12,<3.0
+  - zarr <3.0|>=3.0.10
   - furo >=2022.09.15
   - pip:
       - git+https://github.com/ncar-xdev/ecgtools

ci/environment-upstream-dev.yml

@@ -12,6 +12,7 @@ dependencies:
   - gcsfs >=2024.12
   - h5netcdf >=0.8.1
   - ipython
+  - itables
   - matplotlib
   - netcdf4 >=1.5.5,!=1.6.1
   - pandas >=2.1.0
@@ -36,7 +37,7 @@ dependencies:
   - scipy
   - xarray-datatree
   - xgcm
-  - zarr >=2.10,<3.0
+  - zarr <3.0|>=3.0.10
   - pip:
       - git+https://github.com/intake/intake.git
       - git+https://github.com/pydata/xarray.git

ci/environment.yml

@@ -14,6 +14,7 @@ dependencies:
   - h5netcdf >=0.8.1
   - intake >=2.0
   - ipython
+  - itables
   - matplotlib
   - netcdf4 >=1.5.5,!=1.6.1
   - pandas >=2.1.0
@@ -34,5 +35,5 @@ dependencies:
   - scipy
   - xarray >=2024.10
   - xarray-datatree
-  - zarr >=2.12,<3.0
+  - zarr <3.0|>=3.0.10
   # - pytest-icdiff

docs/source/how-to/use-catalogs-with-assets-containing-multiple-variables.md

@@ -12,7 +12,7 @@ kernelspec:
 By default, `intake-esm` assumes that the data assets (files) contain a single variable (e.g. `temperature`, `precipitation`, etc..). If you have multiple variables in your data files, intake-esm requires the following:
 
 - the `variable_column` of the catalog must contain iterables (list, tuple, set) of values (e.g. `['temperature', 'precipitation']`).
-- the user must provide converters with appropriate functions for parsing values in the `variable_column` (and/or any other column with iterables) into iterables when loading the catalog. There are two ways to do this with the `open_esm_datastore` function: either pass the converter functions directly through the `read_csv_kwargs` argument, or specify the columns in `columns_with_iterables` parameter. The latter is a shortcut for the former. Both are demonstrated below.
+- the user must provide converters with appropriate functions for parsing values in the `variable_column` (and/or any other column with iterables) into iterables when loading the catalog. There are two ways to do this with the `open_esm_datastore` function: either pass the converter functions directly through the `read_kwargs` argument, or specify the columns in `columns_with_iterables` parameter. The latter is a shortcut for the former. Both are demonstrated below.
 
 ## Inspect the catalog
 
@@ -40,7 +40,7 @@ dask.config.set(scheduler='single-threaded')
 
 cat = intake.open_esm_datastore(
     "multi-variable-catalog.json",
-    read_csv_kwargs={"converters": {"variable": ast.literal_eval}},
+    read_kwargs={"converters": {"variable": ast.literal_eval}},
 )
 cat
 ```
@@ -63,13 +63,21 @@ cat.esmcat.has_multiple_variable_assets
 
 ## Search for datasets
 
-The search functionatilty works in the same way:
+The search functionality works in the same way:
 
 ```{code-cell} ipython3
 cat_subset =cat.search(variable=["O2", "SiO3"])
 cat_subset.df
 ```
 
+### Interactively search the catalog
+
+We can also use the `interactive` attribute of a catalog to interactively search the catalog. This will not save any searches, but allows you to explore the catalog in a quick and intuitive way.
+
+```{code-cell} ipython3
+cat.interactive
+```
+
 ## Load assets into xarray datasets
 
 When loading the data files into xarray datasets, `intake-esm` will load only **data variables** that were requested. For example, if a data file contains ten data variables and the user requests for two variables, intake-esm will load the two requested variables plus necessary coordinates information.
@@ -81,7 +89,7 @@ dsets
 
 ## Why does `intake.open_esm_datastore` need the `columns_with_iterables` parameter?
 
-Why does intake `intake.open_esm_datastore` need the `columns_with_iterables` argument when we can achieve the same functionality with just `read_csv_kwargs`? Intake facilitates writing YAML descriptions of catalogs that can be opened with `intake.open_catalog`. These YAML descriptions include the information required to open the catalog: things like the catalog driver (`intake_esm.core.esm_datastore` in our case) and the arguments to pass to the driver to open the catalog. They can be included as entries in other catalogs enabling features like [catalog nesting](https://intake.readthedocs.io/en/latest/catalog.html#catalog-nesting). However, intake does not support Python function arguments like those we provided to `read_csv_kwargs` above so if we want a functional intake YAML description of an intake-esm catalog with multi-variable assets we need to use the `columns_with_iterables` argument instead. You can return an intake YAML description of an `esm_datastore` as follows:
+Why does intake `intake.open_esm_datastore` need the `columns_with_iterables` argument when we can achieve the same functionality with just `read_kwargs`? Intake facilitates writing YAML descriptions of catalogs that can be opened with `intake.open_catalog`. These YAML descriptions include the information required to open the catalog: things like the catalog driver (`intake_esm.core.esm_datastore` in our case) and the arguments to pass to the driver to open the catalog. They can be included as entries in other catalogs enabling features like [catalog nesting](https://intake.readthedocs.io/en/latest/catalog.html#catalog-nesting). However, intake does not support Python function arguments like those we provided to `read_kwargs` above so if we want a functional intake YAML description of an intake-esm catalog with multi-variable assets we need to use the `columns_with_iterables` argument instead. You can return an intake YAML description of an `esm_datastore` as follows:
 
 ```{code-cell} ipython3
 cat.name = "my-esm-catalog"

docs/source/reference/esm-catalog-spec.md

@@ -85,11 +85,44 @@ The column names can optionally be associated with a controlled vocabulary, such
 
 An assets object describes the columns in the CSV file relevant for opening the actual data files.
 
-| Element            | Type   | Description                                                                                                                                                                                                            |
-| ------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| column_name        | string | **REQUIRED.** The name of the column containing the path to the asset. Must be in the header of the CSV file.                                                                                                          |
-| format             | string | The data format. Valid values are `netcdf`, `zarr`, `opendap` or `reference` ([`kerchunk`](https://github.com/fsspec/kerchunk) reference files). If specified, it means that all data in the catalog is the same type. |
-| format_column_name | string | The column name which contains the data format, allowing for variable data types in one catalog. Mutually exclusive with `format`.                                                                                     |
+| Element            | Type   | Description                                                                                                                                                                                                                              |
+| ------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| column_name        | string | **REQUIRED.** The name of the column containing the path to the asset. Must be in the header of the CSV file.                                                                                                                            |
+| format             | string | The data format. Valid values are `netcdf`, `zarr`, `zarr2`, `zarr3`, `opendap` or `reference` ([`kerchunk`](https://github.com/fsspec/kerchunk) reference files). If specified, it means that all data in the catalog is the same type. |
+| format_column_name | string | The column name which contains the data format, allowing for variable data types in one catalog. Mutually exclusive with `format`.                                                                                                       |
+
+````{note}
+ Zarr v3 is built on asynchronous operations, and requires `xarray_open_kwargs` to contain the following dictionary fragment:
+ ```python
+xarray_open_kwargs = {
+    "storage_options" : {
+        "remote_options" : {
+            "async": true,
+            ...
+        },
+        ...
+    },
+   ...
+}
+ ```
+
+In contrast, Zarr v2 is synchronous and instead requires:
+
+ ```python
+xarray_open_kwargs = {
+    "storage_options" : {
+        "remote_options" : {
+            "async": false,
+            ...
+        },
+        ...
+    },
+   ...
+}
+ ```
+````
+
+If `zarr2` or `zarr3` is specified in the `format` field, the `async` flag will be set automatically. If you specify `zarr` as the format, you must set the `async` flag manually in the `xarray_open_kwargs`.
 
 ### Aggregation Control Object