Skip to content

Conversation

mmccrackan
Copy link
Contributor

@mmccrackan mmccrackan commented Aug 22, 2025

Work in progress. Sort of an update to #1156. Does the following:

  1. Switches to per-group parallelization for preprocess_tod and multilayer_preprocess_tod.
  2. Reworks preprocess_tod and multilayer_preprocess_tod to rely more on preproc_or_load_group since it does many of the same things.
  3. Progress bar for preprocess_tod and multilayer_preprocess_tod.
  4. Start adding jobdb support for both preprocess_tod and multilayer_preprocess_tod following similar implementation to record_qa.
  5. Adds some standardized error messages in a class which can be input into a preprocessing error db.
  6. Some more cleanup of functions and docstrings in preprocess_util.

Might split into multiple PRs if possible. Will need to adjust some things to make sure it doesn't break things downstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant