upgrade to odc>1.9 #195

emmaai · 2025-06-13T01:44:15Z

As the tile. Current develop is tagged as develop_1.8 for record.

integrated changes made by @Ariana-B on the branch integrate_1.9 wrt update on odc.*>1.9
shuffled numexpr related functionality back to odc-algo
adopted native_loading from odc-algo atm, further discussion on shuffling the code again

Note that:

tests on code run code check should pass
tests on the older docker image statistician test and push won't as it has old dependencies.

for more information, see https://pre-commit.ci

Co-authored-by: Emma Ai <emma.ai@ga.gov.au>

updates: - [github.com/adrienverge/yamllint.git: v1.37.0 → v1.37.1](https://github.com/adrienverge/yamllint.git/compare/v1.37.0...v1.37.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

codecov · 2025-06-13T02:27:32Z

Codecov Report

Attention: Patch coverage is 83.01486% with 80 lines in your changes missing coverage. Please review.

Project coverage is 81.50%. Comparing base (d208092) to head (2245438).
Report is 4 commits behind head on develop.

Files with missing lines	Patch %	Lines
odc/stats/io.py	68.42%	48 Missing ⚠️
odc/stats/model.py	73.80%	11 Missing ⚠️
odc/stats/_gjson.py	45.45%	6 Missing ⚠️
odc/stats/tasks.py	73.68%	5 Missing ⚠️
odc/stats/proc.py	66.66%	3 Missing ⚠️
odc/stats/_grouper.py	96.00%	2 Missing ⚠️
odc/stats/_text.py	91.66%	1 Missing ⚠️
odc/stats/plugins/lc_ml_treelite.py	80.00%	1 Missing ⚠️
odc/stats/plugins/lc_tf_urban.py	80.00%	1 Missing ⚠️
tests/__init__.py	50.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #195      +/-   ##
===========================================
+ Coverage    81.47%   81.50%   +0.03%     
===========================================
  Files           50       53       +3     
  Lines         4604     4824     +220     
===========================================
+ Hits          3751     3932     +181     
- Misses         853      892      +39

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

odc/stats/model.py

SpacemanPaul

A couple of minor tweaks and/or clarifications requested. Otherwise looking great!

tests on the older docker image statistician test and push won't as it has old dependencies.

It should be possible to get this working before merge, and doing so would make me feel a bit more comfortable - we do always run stats in a docker container after all. Happy to have different docker image tags for 1.8 and 1.9 branches if that helps.

odc/stats/io.py

odc/stats/model.py

emmaai · 2025-06-18T05:17:40Z

A couple of minor tweaks and/or clarifications requested. Otherwise looking great!

tests on the older docker image statistician test and push won't as it has old dependencies.

It should be possible to get this working before merge, and doing so would make me feel a bit more comfortable - we do always run stats in a docker container after all. Happy to have different docker image tags for 1.8 and 1.9 branches if that helps.

It is possible but it'll defy the purpose of failing it in the long run. the docker image won't be published/pushed if failure after merging, which guarantees:

either stats or dependencies failure would stop docker image into production
the dependencies-only update will guarantee old version of stats is till working (hence failed in this pr as old version of stats not working anymore)

SpacemanPaul · 2025-06-18T06:22:25Z

A couple of minor tweaks and/or clarifications requested. Otherwise looking great!

tests on the older docker image statistician test and push won't as it has old dependencies.

It should be possible to get this working before merge, and doing so would make me feel a bit more comfortable - we do always run stats in a docker container after all. Happy to have different docker image tags for 1.8 and 1.9 branches if that helps.

It is possible but it'll defy the purpose of failing it in the long run. the docker image won't be published/pushed if failure after merging, which guarantees:

OK I was confused about how the github actions work. Agreed.

Ariana-B · 2025-06-19T02:16:52Z

odc/stats/model.py

+                dataset.metadata_doc, skip_validation=True
+            )
+            # it will grab all useful input dataset preperties
+            dataset_assembler._inherit_properties_from(


Is this still necessary now that we're no longer passing dataset_assembler the dataset doc?

the serialised dataset doc is still passed to dataset_assembler to grab some properties. I broke the same functionality in 3 parts,

validate classifier 401

inherit properties from source 418

record lineage 422

But doesn't this mean that we're treating the one dataset as both the source the derived dataset? Or does the distinction not particularly matter in this case

SpacemanPaul · 2025-06-19T05:31:03Z

odc/stats/model.py

+            tree = LineageTree.from_eo3_doc(ds.metadata_doc)
+            lineage |= (
+                tree.child_datasets() if tree.child_datasets() else {tree.dataset_id}
            )


Oh yeah, that's much better.

Ariana Barzinpour and others added 23 commits April 8, 2025 05:10

update to 1.9 and use odc.geo; most tests passing

d2ab2fc

expect dataset sources as list

7209aeb

update test

50904be

set XYSCALE=None explicitly to use odc-geo warp

b5de1d6

update reqs

cf1cf79

tests requirement with odc>1.9

06ac710

change time to timezone aware

95d4c2d

relocate loading from odc-algo

fec3e81

delete cached stage periodically

149444c

rename docker image workflow yaml

f8136eb

update basics with pass

9aed4f6

update tflite runtime name

27e344f

add back stats and install newest numexpr

1f973d6

switch from ubuntu to debian

0828840

remove odc-stats

f368d44

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b78473

for more information, see https://pre-commit.ci

fix plugin import

5807a19

upgrade to odc>1.9

695b7cb

update integration test

f7fca14

add indexing test (#191)

9c25437

Co-authored-by: Emma Ai <emma.ai@ga.gov.au>

update dependencies in test

b020172

remove dask client from unit test

02b0106

Emma Ai added 4 commits June 16, 2025 04:43

update odc-algo hash

9451e14

update env variable with datacube>1.9

9a33abe

remove pin on sqlalchemy

332146b

lower pin numexpr

8194168

emmaai requested review from SpacemanPaul and Ariana-B June 16, 2025 05:13

emmaai force-pushed the develop_1.9 branch from d33d575 to 493bbe5 Compare June 17, 2025 05:00

update dependencies version

92c45e6

emmaai force-pushed the develop_1.9 branch from 493bbe5 to 92c45e6 Compare June 17, 2025 05:02

Emma Ai added 5 commits June 17, 2025 05:20

remove expr_eval

c3ba410

install numexpr from pypi

35ff595

fix typos

385153e

update dependencies in test

9bd0f48

more typos

e8d2f52

emmaai marked this pull request as ready for review June 17, 2025 05:38

Ariana-B reviewed Jun 17, 2025

View reviewed changes

odc/stats/model.py Outdated Show resolved Hide resolved

odc/stats/model.py Outdated Show resolved Hide resolved

Emma Ai added 2 commits June 18, 2025 04:21

save and retrieve info on fused product from properties explicitly

ae83fff

always some typos

b51a8a3

SpacemanPaul requested changes Jun 18, 2025

View reviewed changes

odc/stats/io.py Outdated Show resolved Hide resolved

odc/stats/model.py Outdated Show resolved Hide resolved

odc/stats/model.py Outdated Show resolved Hide resolved

update typing to 3.10

3c91b5b

emmaai requested a review from SpacemanPaul June 19, 2025 00:01

unhacking lineage and metadata assemble

2245438

emmaai force-pushed the develop_1.9 branch from f928250 to 2245438 Compare June 19, 2025 00:02

emmaai requested a review from Ariana-B June 19, 2025 00:07

Ariana-B reviewed Jun 19, 2025

View reviewed changes

SpacemanPaul approved these changes Jun 19, 2025

View reviewed changes

emmaai merged commit 1ec34db into develop Jun 19, 2025
8 of 10 checks passed

emmaai deleted the develop_1.9 branch June 19, 2025 05:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

upgrade to odc>1.9 #195

upgrade to odc>1.9 #195

Uh oh!

emmaai commented Jun 13, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jun 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

SpacemanPaul left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emmaai commented Jun 18, 2025 •

edited

Loading

Uh oh!

SpacemanPaul commented Jun 18, 2025

Uh oh!

Ariana-B Jun 19, 2025

Uh oh!

emmaai Jun 19, 2025 •

edited

Loading

Uh oh!

Ariana-B Jun 19, 2025

Uh oh!

SpacemanPaul Jun 19, 2025

Uh oh!

Uh oh!

Uh oh!

upgrade to odc>1.9 #195

upgrade to odc>1.9 #195

Uh oh!

Conversation

emmaai commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

SpacemanPaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emmaai commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SpacemanPaul commented Jun 18, 2025

Uh oh!

Ariana-B Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

emmaai Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ariana-B Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

SpacemanPaul Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

emmaai commented Jun 13, 2025 •

edited

Loading

codecov bot commented Jun 13, 2025 •

edited

Loading

emmaai commented Jun 18, 2025 •

edited

Loading

emmaai Jun 19, 2025 •

edited

Loading