Containerization of PODs

[tsjackson-noaa]

@aradhakrishnanGFDL has expressed interest in containerizing POD execution. I'm starting this discussion thread to document our progress on the topic:

• Containerization functionality will be useful in the short run and probably necessary in the long run: we'd like the framework to be able to run (hypothetical) diagnostics for which the primary method of installation is via container, and we'll also need the framework to run (less hypothetical) legacy scripts that are needed but which we lack the resources to maintain -- I'd envision containerizing a snapshot of the script's dependencies in that case.
• We currently do dependency management with conda, which could be said to occupy a middle ground between python virtualenvs and full containerization. We've already hit issues where conda didn't guarantee complete cross-platform reproducibility (XQuartz is a required dependency for NCL-based PODs on Mac OS #65, Consider pinned versions in conda environments more carefully #86), and containerization would conclusively fix these.

[tsjackson-noaa]

Jotting down some background remarks on the code's organization -- I'm optimistic that we should be able to accommodate containerized execution without having to tear up and redesign things:

• The current mechanism for handling PODs' dependencies is that
  1: PODs list their third-party libraries and other dependencies as part of their settings file.
  2: This setting is read by an EnvironmentManager object, which has the (loosely defined) responsibility of ensuring that those dependencies are made available to the POD when it's run. For example, the default EnvironmentManager (CondaEnvironmentManager) uses user-installed conda environments to provide the dependencies. At runtime, it decides which of these environments to use, and inserts a shell command to activate that environment before the POD itself is run.
  3: In v3alpha3, the responsibility for actually running the POD has been split off into a RuntimeManager class. Currently the only implementation just runs each POD in a separate local subprocess and checks the return code.
• The first change would be to require more explicit versioning of dependencies in step 1, eg. by following conda's YAML spec for environments (example). We'd like each POD to provide a list at this level of detail (either their own unique dependencies, or pointing to a shared set of "typical/base" libraries we provide).
• The second step would be to write an EnvironmentManager that builds the container recipe from those specifications (either on-the-fly, or as an optional part of the MDTF package installation), and a RuntimeManager that interfaces with docker to run the containers. To get a working proof of concept, we could start by just feeding the existing conda environments into a miniconda docker image. This is where I need to get my hands dirty and learn more about the details, but I'd imagine the recipe would be very similar to our existing Travis CI configuration or the work that @wrongkindofdoctor is doing to port that to Github Actions.