preparing a CMOR demo with a coordinate having character string entries

[gleckler1]

@mauzey1 @durack1 @taylor13
With the goal of setting up a cmor demo for obs4MIPs, I am trying to reproduce a CMIP6 variable that has dimensions [latitude basin time], with "basin" being an array of character strings. A copy of the input file (hfbasin_Omon_UKESM1-0-LL_ssp370_r5i1p1f2_gnz_201501-204912.nc) used in this demo is in the same GH directory as the test.py script. It appears to have been prepared with CMOR as a contribution to CMIP6.

The difficulty I am having is almost certainly with data type of sector
cmorBasin = cmor.axis("basin", coord_vals=sector[:],units="")

but there is no error until the cmor.write (below) and no other info is provided.

Any hints???

(xarray-cmor-v20250616) u7510@perlmutter:login14:~/git/obs4MIPs-cmor-tables/demo/demo-site_colletion> python -i test.py
Traceback (most recent call last):
File "/global/u1/u/u7510/git/obs4MIPs-cmor-tables/demo/demo-site_colletion/test.py", line 42, in
cmor.write(varid,d,len(time))
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/global/homes/u/u7510/.conda/envs/xarray-cmor-v20250616/lib/python3.13/site-packages/cmor/pywrapper.py", line 799, in write
raise Exception("Error could not convert data to a numpy array")
Exception: Error could not convert data to a numpy array

[mauzey1]

@gleckler1 There were two issues that I found in the demo.

On the line that sets the list of dimensions, the order should be [cmorTime, cmorBasin, cmorLat].

The error that you are encountering is caused by passing a Xarray DataArray rather than a NumPy array to cmor.write. To fix the error, either pass d.values or the values variable to cmor.write.

After applying the above fixes, I was getting a segfault from cmor.write. I will investigate what is causing the segfault.

[gleckler1]

Thanks, @mauzey1, I have fixed the two issues you have identified and reproduced the segfault. I suspect it is related to cmorBasin but unsure how that can be resolved.

[mauzey1]

@gleckler1 The segfault is being caused by NaN values in the variable data.

Adding the line d = np.where(np.isnan(d), 1.0e20, d) to remove the NaNs will remove the segfault.

[gleckler1]

@mauzey1 That is pretty fabulous, thank you! The last test I need to construct before preparing the demo involves introducing another dimension -> hfbasin(time, basin, lat,lon). Do you think CMOR can accommodate inclusion of
with this added dimension, i.e., still permit introduction of char sector(basin, strlen) as a variable? Thank you again!!

[taylor13]

Just curious. Why do you need to indicate "basin" if you provide "lat" and "lon" for each reported value. Wouldn't it be better to simply record the "basin" as a function of lat and lon as a separate variable? I know we did this in CMIP5 and assigned an integer index to each of about 10 basins. The fixed field then was just one integer per grid cell (identified by longitude and latitude). Not sure what was done in CMIP6.

[gleckler1]

@taylor13 Good question. Still prototyping. The ultimate purpose here is for site data akin to cf-sites. This is one of the REF short-listed requirements on obs4MIPs. An example dataset is FluxNet... these and other sites are being gathered for use in obs4MIPs. The ARM sites acronyms chosen by ARM will supersede the acronyms created by Fluxnet.

[durack1]

@gleckler1 The segfault is being caused by NaN values in the variable data.

@mauzey1 another segfault! It'd be great to augment the code to catch yet another segfault case. @gleckler1 thanks for pushing CMOR in a direction we'd not been testing!

[gleckler1]

@mauzey1 @taylor13 @durack1 I've tried to generalize inclusion of the "dimension with strings" to 2D (lon,lat). Now working with "sites" rather than "basin", the roadblock I've hit is the requirement for lon and lat to be monotonic. Is there anyway that can be relaxed?

I've added "sites" to the coordinate.json, and a new variable to the Amon table with the desired dimensions.

I've tried constructing axes the usual way and also creating a grid, but in both cases the monotonic dimensions is a problem.

Any hints?

C Traceback:
! In function: cmor_check_monotonic
! called from: cmor_treat_axis_values
! called from: cmor_axis
!!!!!!!!!!!!!!!!!!!!!!!!!
! Error: axis y_deg (table: grids) has non monotonic values:
! -28.239500, 47.116670 and -22.283000
!!!!!!!!!!!!!!!!!!!!!!!!!

[mauzey1]

I've identified where the segfault is happening in CMOR. It's happening where it is locating NaN values within the variable data by axis values.

cmor/Src/cmor_variables.c

Lines 2880 to 2891 in 810e1f2

	for (j = 0; j < avar->ndims; j++) {
	cmor_axis_t *pAxis;
	pAxis = &cmor_axes[avar->axes_ids[j]];
	double val = 0.f;
	if (pAxis->values != NULL) {
	val = pAxis->values[nan_counter[j]];
	} else {
	val = time_vals[nan_counter[j]];
	}
	msg_len += snprintf(NULL, 0, " %s: %lu/%.5g",
	pAxis->id, nan_counter[j], val);
	}

The segfault occurs when one of the axis definitions is null, which should be the "basin" axis. It then tries to access thetime_vals array passed from cmor.write, which is also null since those weren't passed with cmor.write in the demo. This could have also happened with checks for minimum and maximum value within the same function.

There are two issues to resolve here.

1. Figure out why the "basin" axis is null.
2. Prevent the time values passed array from being accessed if it is null.

[gleckler1]

@mauzey1 Can you confirm the cmor_check_monotonic can not be turned off for the case described above?

[mauzey1]

@gleckler1 Yes, you can't turn off cmor_check_monotonic.

[gleckler1]

@mauzey1 That is very encouraging. How does one do that? It looks like cmor_axis[coord_vals] requires the axes to be monotonic.

[mauzey1]

@gleckler1 In the case of using the site axis, you can follow what is performed in test_python_cfmip_site_axis_test.py. Specifically, using the site axis for the grid axis with the latitude and longitude arrays from the file.

cmor/Test/test_python_cfmip_site_axis_test.py

Lines 67 to 88 in 810e1f2

	saxis_id = cmor.axis('site', units='1', coord_vals=[1, 2, 3])
	print('ok: created site axis', saxis_id)
	zaxis_id = cmor.axis(
	'hybrid_height',
	units='m',
	coord_vals=[1.0],
	cell_bounds=[
	0.0,
	2.0])
	print('ok: created height axis', zaxis_id)
	# Create zfactors for b and orog for hybrid height axis.
	# Where do these get used, if anywhere?
	bfact_id = cmor.zfactor(zaxis_id, 'b', '1', [zaxis_id], 'd', zfactor_values=[1.0],
	zfactor_bounds=[0.0, 2.0])
	print('ok: created b zfactors')
	# Create grid object to link site-dimensioned variables to (lat,long).
	# Need to make CMIP6_grids the current MIP table for this to work.
	table_id = cmor.load_table('CMIP6_grids.json')
	gaxis_id = cmor.grid([saxis_id], site_lats, site_lons)

[taylor13]

Not sure this is relevant, but a dimension in netCDF must be either an index or have a numerical coordinate values. If you want a label, it has to be associated with the dimensions through a "coordinates" attribute. I'll try to read the above when I have time.