Skip to content

Conversation

omad
Copy link
Member

@omad omad commented Sep 5, 2025

I was setting up a local database for testing Explorer, and cubedash-gen was throwing stack traces left right and center at me.

It only occurred with multiple jobs, which is the default.

I suspected it's something we're doing wrong with structlog, on the structlog Performance page it says to not use cache_logger_on_first_use=True if you're passing loggers around with multiprocessing, which is what cubedash-gen uses for running in parallel.

But, that didn't fix it.

What did fix it was changing the line that was the top of the exception to take a kwarg instead of an arg. I don't know why this works, but, it does, and I've spent too long debugging!

2025-09-05 15:22:22 [error    ] product.error                  product=abares_clum_2023
Traceback (most recent call last):
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/generate.py", line 136, in generate_report
    result, updated_summary = store.refresh(
                              ^^^^^^^^^^^^^^
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/summary/_stores.py", line 1289, in refresh
    rv = self.refresh_product_extent(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/summary/_stores.py", line 488, in refresh_product_extent
    self._refresh_product_regions(product)
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/summary/_stores.py", line 500, in _refresh_product_regions
    log.info("refresh.regions.inserted", list(result))
  File "/Users/aye011/dev/odc/datacube-explorer/.venv/lib/python3.12/site-packages/structlog/_native.py", line 146, in meth
    return self._proxy_to_logger(name, event % args, **kw)
                                       ~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting
2025-09-05 15:22:22 [error    ] product.error                  product=abares_clum_2020
Traceback (most recent call last):
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/generate.py", line 136, in generate_report
    result, updated_summary = store.refresh(
                              ^^^^^^^^^^^^^^
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/summary/_stores.py", line 1289, in refresh
    rv = self.refresh_product_extent(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/summary/_stores.py", line 488, in refresh_product_extent
    self._refresh_product_regions(product)
  File "/Users/aye011/dev/odc/datacube-explorer/cubedash/summary/_stores.py", line 500, in _refresh_product_regions
    log.info("refresh.regions.inserted", list(result))
  File "/Users/aye011/dev/odc/datacube-explorer/.venv/lib/python3.12/site-packages/structlog/_native.py", line 146, in meth
    return self._proxy_to_logger(name, event % args, **kw)
                                       ~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting

📚 Documentation preview 📚: https://datacube-explorer--829.org.readthedocs.build/en/829/

@pjonsson
Copy link
Contributor

pjonsson commented Sep 5, 2025

Your backtrace looks like hynek/structlog#258, so we should probably fix all of these if there are more of them somewhere in the codebase.

I'm inclined to agree with your multiprocessing assessment, but I'm guessing the test error is caused by cache_logger_on_first_use=False rather than using the keyword argument. Can you split out the keyword fix to a separate PR so we get that bug fix merged immediately and then we ponder the correctness of caching the logger separately?

@omad
Copy link
Member Author

omad commented Sep 5, 2025

Okay, putting in just the fix for the positional/keyword argument case. I looked and couldn't find any more of the same.

The test failure looks the same as the pin that's keeping click < 8.2.

Will look into it later.

Copy link

codecov bot commented Sep 5, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.35%. Comparing base (a6f12ce) to head (5d44102).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #829   +/-   ##
========================================
  Coverage    84.35%   84.35%           
========================================
  Files           35       35           
  Lines         4244     4244           
  Branches       537      537           
========================================
  Hits          3580     3580           
  Misses         468      468           
  Partials       196      196           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pjonsson pjonsson changed the title Work-around for crashes in cubedash-gen Fix crash caused by faulty call to logger Sep 6, 2025
@pjonsson pjonsson merged commit 1906ab4 into develop Sep 6, 2025
20 of 22 checks passed
@pjonsson pjonsson deleted the gen-crashes branch September 6, 2025 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants