MDTF diagnostics in the cloud

[tsjackson-noaa]

This discussion thread is to capture the work that would be needed to run a cloud-hosted version of the MDTF diagnostics, as suggested by @aradhakrishnanGFDL and @jkrasting. I think this project would yield many benefits and, more to the point, is very feasible given the current capabilities of the code.

• Deployment: at least initially, I believe it would make the most sense to deploy the entire package as a single container, rather than needing to coordinate handing execution off to multiple, separate containerized PODs (discussion Containerization of PODs #94).
  • This may run into ceilings in terms of available CPU and RAM on the instances we get, though. For this reason, in the proof-of-concept we may want to restrict execution to a single POD per invocation of the package.
  • For the dockerfile itself I'm assuming we can leverage the work @wrongkindofdoctor has done with CI; the image could be based off of any miniconda image.
  • Loading the obs data may pose a difficulty if incoming FTP connections are blocked or unreliable; presumably this could be handled by hosting a copy of the obs data in the cloud as well.
• Data query: data queries in v3alpha3 are done against a pandas dataframe, similar to what's used by intake-esm. In the current code, this dataframe is assembled on-the-fly by crawling the directory of model data files, but any cloud data catalog that presents this interface should work without the need to adapt the query logic.
  The technical caveat is that the catalog, as it's currently implemented, is populated with objects rather than their serializations, e.g. DateRanges instead of the string '1990-2010'. Additional code will be needed to deserialize catalog entries when they're being compared to variable data in the query.
• Data fetch: in v3alpha3, data fetch logic is separated from the data query logic. The data query uses intake-esm, and the preprocessor uses dask, both of which use fsspec as a layer for remote filesystem access. In particular, fsspec supports S3, GCS and Azure storage backends -- see discussion Use fsspec as a general data fetch back end #102.
  • I think that only some cleanup of the existing code would be needed to enable fetching data from cloud storage (the technical detail being that we still load data files manually in the preprocessor, rather than doing so directly from the query with intake-esm, because we need to do experiment selection logic in between querying the catalog and fetching the data.)
  • Handling of the remote data location has been compartmentalized into methods, and can accommodate switching from filesystem paths to S3 URLs, etc.
• Data processing: in v3alpha3, all model data files pass through the preprocessor. Since the preprocessor uses xarray for all data handling, reading data in xarray-compatible zarr format would only require modifications to the dataset loading methods.
  Note that, regardless of input format, the output of the preprocessor would still be a set of netcdf files, as this is what all PODs are currently written to accept. These can be written anywhere on local storage, e.g. to a temp directory, but we may run into storage limitations on the instance. Using dask to hand off in-memory Datasets from the framework to PODs is definitely a feature we want in the future, but beyond the scope of this exercise.
• User interface, i.e. how the user would start the package and view its output. Details of how we set this up depend on the conventions we're using for the user's cloud access -- I'd especially welcome suggestions here. Output of the package is a self-contained set of static HTML files, which is well-suited to being served by the cloud provider.
  For concreteness, and by analogy to what I've seen with Pangeo, my initial proposal would be doing this via a
• Jupyter notebook user interface: This is a feature that's highly desirable, independent of the cloud use case considered here, and should probably be spun off into its own discussion/issue.
  • The CLI parsing code has been pretty well organized, so only minor cleanup would be needed to let the user invoke the package based on a dict of configuration options instead of command-line flags.
  • We already have code for querying the available PODs, which could be used for letting the user select PODs to run interactively.
  • Invoking the query logic to provide interactive feedback on whether the user's chosen source of model data provides all the variables needed by the user's chosen POD would be desirable in its own right as a debugging feature.
  • Formatting and displaying the POD output probably poses the greatest hurdle out of all the factors listed here. POD plots are converted to bitmap files, and presumably Jupyter has hooks available for displaying a graphics file from a known location in the notebook according to the user's current configuration. The real difficulty is that a POD may generate 20+ plots, and most plots aren't meaningful without comparison to a corresponding plot of observational data. Since the jupyter interface is running in a browser as it is, the most sensible thing to do would probably be to "punt" and load the POD's HTML output page in a new tab, rather than trying to cram all its output into a single notebook cell.