Add new compsets and namelists for km-scale ELM simulation

[daliwang]

The new features support scalable, km-scale E3SM land modeling (using the data atmosphere model) across various computational resources, from a single core to up to 2400 nodes on leadership-class supercomputers.

Wang, D., Wang, C., Cao, Q., Krishna, J., Wu, D., Zheng, W., Schwartz, P., Yuan, F., Mohror, K., & Thornton, P. (2025). Scaling Ultrahigh-Resolution E3SM Land Model for Leadership-Class Supercomputers, IEEE Symposium on Cluster, Cloud, and Internet Computing 2025, TCSC SCALE Challenge finalist.

Tested and passed e3sm_land_developer tests on Frontier using craygnu compiler.

[BFB]

[rljacob]

How big are some of these new data files? Are they in the correct format (CDF5)?

[daliwang]

All the dataset can be softlinks to small datasets, they are in CDF5 format.

[peterdschwartz]

@rljacob the files are custom for each domain. For the test case using these forcings (21 gridcells), the forcings are ~32KB and the surface dataset is ~1GB.

For the file format, ncdump -k says netCDF-4, so I will need to convert them

[bishtgautam]

How about naming these datasets as DaymetGSWP3 and DaymetERA5?

[daliwang]

Good ideas.

[bishtgautam]

How about deleting these lines?

[daliwang]

These two lines are needed for large-scale experiments, after discussion with Jayesh.

[bishtgautam]

How about deleting this line?

[daliwang]

These are related to a small test case that Peter is used for kmELM evaluation.

[bishtgautam]

Examining the data on NERSC, it is a highly customized DATM dataset created specifically for the study. e.g. the dataset has only 24 grd cells. Am I missing something here?

ncdump -h /global/cfs/cdirs/e3sm/inputdata/atm/datm7/atm_forcing.datm7.km.1d/MOF21_clmforc.Daymet4.1km.1d.WIND.2014-12.nc
netcdf MOF21_clmforc.Daymet4.1km.1d.WIND.2014-12 {
dimensions:
	time = 248 ;
	ni = 21 ;
	nj = 1 ;
variables:
	int gridID(nj, ni) ;
		gridID:long_name = "gridId in the NA domain" ;
		gridID:decription = "Covers all land and ocean gridcells, with #0 at the upper left corner of the domain" ;
	float time(time) ;
		time:long_name = "observation time in UTC" ;
		time:units = "days since 2014-12-01 00:00:00" ;
		time:calendar = "noleap" ;
	double LATIXY(nj, ni) ;
		LATIXY:units = "degrees_north" ;
		LATIXY:long_name = "latitude coordinate" ;
		LATIXY:standard_name = "latitude" ;
	double LONGXY(nj, ni) ;
		LONGXY:units = "degrees_east" ;
		LONGXY:long_name = "longitude coordinate" ;
		LONGXY:standard_name = "longitude" ;
	float WIND(time, nj, ni) ;
		WIND:long_name = "wind at the lowest atm level" ;
		WIND:units = "m/s" ;

// global attributes:
		:title = "WIND(2014-12) creted from /gpfs/wolf2/cades/cli185/proj-shared/wangd/Forcing_2014/daymet4_2d/ on 04022024" ;
}

Should we support such a custom dataset in the main branch?

[peterdschwartz]

@bishtgautam You are correct that it is just a customized forcing data set (interpolation of forcings are already done -@daliwang can better comment on what else is done). The main benefit i see with using these datasets for testing is to remove the bottleneck for reading and interpolating the forcings.

For example, the test i added which runs the 21 sites for 1 year only took 78s. Contrast that to the SMS_Ly5_P1x1.1x1_smallvilleIA.IELMCNCROP.pm-cpu_intel.elm-per_crop test which takes 20-30 minutes to complete, but only ~3 minutes of the time is in ELM. Switching that test to use a customized forcing data set, should significantly bring the test runtime down.

I just created this test for the MOFlux sites as an example, and I think it would be great to discuss which region(s) would be best to include in testing. I'm fine to remove the test from the nightly test suites from this PR -- the testmod would be nice to keep for easy case generation though not that big of a deal.

I do think we should merge the compset definitions as they can be very useful and even necessary for adequate I/O performance for 1km runs

[bishtgautam]

In my opinion, if we add a new DATM dataset, we should be able to run a simulation that is global or regional (as in this case, because it is DAYMET). But that isn't possible here.

If the purpose of these files is to improve testing and/or provide an example of how to speed up offline ELM simulation, why not use the user_data.streams.txt file to correctly point to these new datasets on ours instead of creating a compset?

[peterdschwartz]

I would definitely prefer to implement this without a compset as this can work with any ELM configuration (well, i used it for combinations of bgc/fates and different nu_com so far). Unfortunately, i am not very familiar with using the user_data.streams.txt. We would just need to put that file in the testmod as is done with namelist files?

[bishtgautam]

Next week, I can convert the new tests to use user_data.streams.txt instead.

[rljacob]

Any progress on this PR?

[rljacob]

converting to draft until this is reimplemented as described.

[daliwang]

Next week, I can convert the new tests to use user_data.streams.txt instead.

I think we could not just simply use the "user_data.streams.txt" to achieve the results and performance, as the default Datm requires grid checking and spatial interpolation, and it is the remain reason that the coupler-bypass was implemented at ORNL. The current new Compset and associate change, maybe we can change to a more common name, solved the unnecessary grid checking/validation/interpolation issues, enabled the large-scale offline ELM simulation at scales.

[peterdschwartz]

@bishtgautam I removed the new land compsets and changed the names of the new datm to DaymetGSWP3 and DaymetERA5.

[rljacob]

@bishtgautam can you approve this?

[rljacob]

@peterdschwartz please start merging this.