Incorrect time_range in catalog causes preprocessor to hang

[emaroon]

Bug Severity

• 1 = Minor problem that does not affect total framework functionality (e.g., computation error in a POD, problem with logging output, or an issue on a single system
• 2 = Major problem that affects overall functionality, but that does not occur for all users (e.g., problems installing the framework with a specific Conda version, a framework option that causes one or more PODs to fail, or missing/incompatible Python modules).
• 3 = Catastrophic problem that occurs frequently for multiple users and/or on multiple systems (e.g.,framework consistently fails to install on multiple systems, or one or more PODs continuously fails after running successfully)

Describe the bug
Preprocessor hangs indefinitely on a variable if the time_range doesn't have the expected format. No errors are thrown, just hanging.

Steps To Reproduce
I used the catalog tool included in the mdtf to create a CMIP-style catalog. The time_range created by that tool has a format like:
1850-01-15 13:00:00.000007-2014-12-15 12:00:00

It appears like the preprocessor cannot handle it - the MDTF hangs indefinitely. If I manually overwrite the catalog to have a time_range that follows this expected format:
1850-01-15-2014-12-15
then everything works.

I traced the hanging to line 1221 in src/preprocessor.py

                if not var.is_static:
                    if "chunk_freq" in cat_subset.df:
                        cat_subset.esmcat._df = self.check_multichunk(cat_subset.df, date_range, var.log)
                    cat_subset.esmcat._df = self.check_group_daterange(cat_subset.df, date_range, var.log) ##HERE I THINK!

Environment
Describe the system environment:

• OS: Derecho/casper: GNU/Linux
• Conda version: conda version : 24.5.0
• branch name and link: branch named cmip_input, https://github.com/emaroon/MDTF-diagnostics/tree/cmip_input

Log information and/or terminal output
The code hangs after this warning, but no bug thrown:

Querying /glade/work/emaroon/mdtf/MDTF-diagnostics/diagnostics/natl_ocean/CMIP_CESM_historical_001.json for variable vsf for case CESM2_historical_r1i1p1f1.
WARNING: /glade/work/emaroon/conda-envs/_MDTF_base/lib/python3.12/site-packages/intake_esm/_search.py:50: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
mask = df[column].str.contains(value, regex=True, case=True, flags=0)

[wrongkindofdoctor]

@bitterbark This looks like an issue with the datetime processing the time metadata in the CESM files. It seems like the starttime should have the format 1850-01-15 12:00:00 (assuming these are monthly means centered mid-month).

These are the lines where the time_range attribute is generated:

MDTF-diagnostics/tools/catalog_builder/parsers.py

Lines 146 to 156 in 6fb7466

	if 'time' in ds.coords:
	time_var = ds.coords['time']
	calendar = None
	if 'calendar' in time_var.attrs:
	calendar = time_var.attrs['calendar']
	if calendar == 'no_leap':
	calendar = 'noleap'
	start_time = cftime.num2date(time_var.values[0], time_var.attrs['units'], calendar=calendar)
	end_time = cftime.num2date(time_var.values[-1], time_var.attrs['units'])
	time_range = start_time.strftime("%Y%m%d:%H%M%S") + '-' + end_time.strftime("%Y%m%d:%H%M%S")
	catalog_info.update({'time_range': time_range})

I'm not sure if the problem is with cftime, the assumptions about the units, the calendar, or some combination thereof, but this where we need to look.

I wouldn't recommend relying on the mdtf builder at this point as the lack of ecgtools support is becoming a problem, too. I would focus identifying which mdtf builder capabilities you want to integrate into the GFDL builder. I can put in some time after hours to help accelerate the process--just don't forget to add my handle so I get notifications.

[jkrasting]

Suggestions are to:

1. Correct the time units in the MDTF/ECG catalog builder (could be tied to CESM's time axis). The issue arises with the CMIP catalog generated from CMORized CESM data. Manually fixing the strings in the catalog works for now.
2. The framework should give a warning rather than hang without feedback to user

[bitterbark]

Agree with the above (2) The framework should give a warning rather than hang without feedback to user

As for (1) Since we are transitioning away from the catalog builder implemented in the MDTF code, we should make sure to test this in the CESM-capable GFDL catalog builder.

A few more notes for clarification about the problem because it looks like they are addressing two related but distinct variables, starttime & time_range

Liz is talking about a time_range (or date_range, in the src/preprocessor code snippet above)
Causes error: 1850-01-15 13:00:00.000007-2014-12-15 12:00:00
No error: 1850-01-15-2014-12-15

Whereas Jess mentions that the MDTF starttime should have the format 1850-01-15 12:00:00

We should clarify if this time_range works (consistent with the starttime format):
1850-01-15 12:00:00-2014-12-15 12:00:00

@emaroon Can you include the json & csv files here that work/don't work?