Skip to content

EAMxx: Update Dev Docs #7163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 2, 2025
Merged

EAMxx: Update Dev Docs #7163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 196 additions & 0 deletions components/eamxx/docs/developer/TPLs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
# Third-party Libraries Required by EAMxx

- First, we note that if you are running EAMxx on a supported machine,
then you can expect that the third-party libraries (TPLs) are
installed, properly configured, and recognized by the EAMxx build system.
- However, if you choose to run and/or develop EAMxx on a local machine,
workstation, or unsupported cluster, then you will likely need to build some
number of the TPLs manually.
- **Note:** The exception to this is if you are running on a machine with
a module system that already provides them--e.g., research institutions
that run E3SM regularly.

## List of TPLs

- **Note:** For this section, we will assume you plan to build EAMxx on a fresh
Linux[^only-linux] system with the standard development tools available.
- E.g., a package manager (`apt`, `yum`), `git`, `make`, `cmake`, etc.

??? Abstract "TPLs You Will (almost certainly) Need to Install or Build Manually"

- [OpenMPI](https://docs.open-mpi.org/en/v5.0.x/installing-open-mpi/quickstart.html)
- Only required for builds employing multi-node parallelism.
- Required for parallel configurations of the netCDF libraries.
- Likely to be provided by your package manager or module system.
- [HDF5](https://www.hdfgroup.org/download-hdf5/)
- High-performance data-wrangling library.
- The link above provides `.deb` and `.rpm` binary packages, though
the page also provides a link to build from source.
- [netCDF-C](https://github.com/Unidata/netcdf-c)
- Used for I/O of simulation data in a format containing descriptive metadata.
- [netCDF-fortran](https://github.com/Unidata/netcdf-fortran)
- [PnetCDF](https://parallel-netcdf.github.io)
- Only required for builds employing multi-node parallelism.

**Build Tips**

- As of 2024, the author found much less resistance configuring the build
for the netCDF libraries using Autoconf, rather than CMake.

??? List "Auxiliary TPLs Your System May Already Have Installed"

The following list of TPLs may be pre-installed on some systems, and are
helpful, if not required, for building the netCDF libraries.
However, in the author's experience they did need to be built on a fresh
installation of Red Hat Enterprise Linux 8.

**Required**

- [LAPACK](https://www.netlib.org/lapack/) or [BLAS](https://www.netlib.org/blas/)
- Alternatively, [OpenBLAS](http://www.openmathlib.org/OpenBLAS/),
which is unlikely to be pre-installed.

**Likely Required**

- [Autoconf](https://www.gnu.org/software/autoconf/)
- [Libtool](https://www.gnu.org/software/libtool/)

**Maybe Required**[^computers-amirite]

- [help2man](https://www.gnu.org/software/help2man/)

## Help & Hints

Building third-party libraries can be a pain.
To help here are some reasonable example scripts to start from that should
help getting the ***TPLs You Will Need to Build*** compiled and working.

??? Example "Basic Configuration Script Examples"

!!! Warning "Disclaimer"

We provide these scripts as demonstrative examples, only.
They will almost certainly not build your libraries successfully if they
are copy/paste-ed as they appear here.
That being said, every hardware/software environment is different and
will display different and sometimes strange behaviors.
So, the best advice we can provide is to begin with something
resembling what is below and respond to the compiler messages
by modifying: environment variables, `PATH` or `[LD_]LIBRARY_PATH`
entries, configuration arguments, and compiler arguments as required.
As long as you do ***all of that*** perfectly, you should be just fine.
:sunglasses:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

didn't render 😞

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to add the emoji specs to the mkdocs.yaml in root

  - pymdownx.emoji:
      emoji_index: !!python/name:material.extensions.emoji.twemoji
      emoji_generator: !!python/name:material.extensions.emoji.to_svg

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not only in components/eamxx/mkdocs.yml

(Also note the yaml vs yml 🤷)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and thanks for the tip! I hadn't noticed since I'd only been building from the EAMxx-root mkdocs.yml 🙄


### Assumptions

- Source code is downloaded to a directory we assign to the variable `${TPL_ROOT}`.
- We build the libraries in a separate ***build*** and ***install***
directory in the root directory of the source code.
- E.g., `${TPL_ROOT}/<lib-name>/{build/,install/}`
- The bash scripts below are placed in the `build/` directory.
- Hence, note that when executing the configure script from the
`build/` directory, the final `../configure [...]` command in the
script (that calls autoconf) is running the `configure` command in
the source directory for that TPL--e.g., `${TPL_ROOT}/<lib-name>/`
- `libtool` is installed and on your `PATH`.
- I.e., `which libtool` returns the location of the executable.

??? Abstract "`configure-hdf5.sh`"

```{.shell .copy}
#!/bin/bash

# add MPI C/C++ compilers to the proper environment variables
mpi_path="${TPL_ROOT}/openmpi/install/bin/"
export CC="${mpi_path}/mpicc"
export CXX="${mpi_path}/mpicxx"

# configure HDF5 build
../configure \
--prefix=${TPL_ROOT}/hdf5/install \
--enable-shared \
--enable-hl \
--enable-parallel \
--with-zlib=/usr/include,/usr/lib
```

??? Abstract "`configure-netcdf-c.sh`"

```{.shell .copy}
#!/bin/bash

# add MPI C/C++ to the proper environment variables
mpi_path="${TPL_ROOT}/openmpi/install/bin"
export CC="${mpi_path}/mpicc"
export CXX="${mpi_path}/mpicxx"

# historically, netcdf can be a little picky to build, so it can help to
# make ABSOLUTELY CERTAIN that the build system knows where HDF5 is
# via the "sledgehammer" approach
hdf5_path="${TPL_ROOT}/hdf5/install"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${hdf5_path}/lib"
export LIBRARY_PATH=${LIBRARY_PATH}:${hdf5_path}/lib
export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:${hdf5_path}/lib/pkgconfig
export CPPFLAGS="-I${hdf5_path}/include"

../configure --prefix=${TPL_ROOT}/netcdf-c/install
```

??? Abstract "`configure-netcdf-fortran.sh`"

```{.shell .copy}
#! /bin/bash

# add MPI Fortran compiler to the proper environment variables
mpi_path="${TPL_ROOT}/openmpi/install/bin"
export FC="${mpi_path}/mpif90"

# local variables, to be used below
export NetCDF_C_ROOT=${TPL_ROOT}/netcdf-c/install
export NETCDF_C_LIBRARY=${NetCDF_C_ROOT}/lib
export NETCDF_C_INCLUDE_DIR=${NetCDF_C_ROOT}/include

# add netcdf-C bits and pieces to the proper paths
export LD_LIBRARY_PATH="${NETCDF_C_LIBRARY}:${LD_LIBRARY_PATH}"
export LIBRARY_PATH="${NETCDF_C_LIBRARY}:${LIBRARY_PATH}"
export PATH=${NetCDF_C_ROOT}/bin:${PATH}

# add netcdf-C include and library dir to the requisite compiler flags
export CPPFLAGS="-I${NETCDF_C_INCLUDE_DIR}"
export LDFLAGS="-L${NETCDF_C_LIBRARY}"

# while we're at it, tell it where to find hdf5, too
export HDF5_ROOT=${TPL_ROOT}/hdf5/install

../configure --prefix=${TPL_ROOT}/netcdf-fortran/install
```

??? Abstract "`configure-pnetcdf.sh`"

```{.shell .copy}
#!/bin/bash

mpi_path="${TPL_ROOT}/openmpi/install/bin"
export CC="${mpi_path}/mpicc"
export CXX="${mpi_path}/mpicxx"
export FC="${mpi_path}/mpif90"

../configure \
--prefix=${TPL_ROOT}/pnetcdf/install \
--enable-shared \
MPICC=mpicc \
MPICXX=mpicxx \
MPIF77=mpifort \
MPIF90=mpifort
```

<!-- ======================================================================= -->

[^only-linux]: We only support build instructions for Linux, though some individuals have previously built EAMxx on MacOS with success.
We would not recommend using EAMxx on Windows, other than via WSL.
However, if you do end up building and running on MacOS or Windows,
please let the developers know or submit a PR!
<!-- markdownlint-disable-next-line MD053 -->
[^computers-amirite]: Computers are weird. While it's by no means required from
a functionality standpoint, a recently-configured RHEL8 system seemed to be
convinced it needed help2man for some reason.
45 changes: 26 additions & 19 deletions components/eamxx/docs/developer/dev_quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ This means that, if you have an account on one of these machines, EAMxx tests
can be run with no extra configuration required.
However, many of the standalone EAMxx tests are able to be run on a
reasonably-equipped laptop or personal workstation.[^but-mac]
This second choice requires some extra configurations but the effort required
This second choice requires some extra configurations[^TPLs-etc] but the effort required
may be worth the potential speedup in development.

## Let's Get Started (Quickly)
Expand All @@ -50,11 +50,11 @@ may be worth the potential speedup in development.
As such, we provide no guarantee that any of these are working code, and
the author would be quite surprised if more than a couple of them run
without edits.
If you should take the time to debug these examples, feel free to submit a
If you take the time to debug these examples, feel free to submit a
PR with a working version, and we are sure some future developer will
appreciate it!

### Running EAMxx via `test-all-eamxx`
## Running EAMxx via `test-all-eamxx`

The quickest method for a new developer to get EAMxx up and running is to use
the automated configure/build/test workflow provided by `scripts/test-all-eamxx`.
Expand All @@ -64,15 +64,27 @@ errors preventing successful running and testing.

Running via `test-all-eamxx` requires ***at minimum*** 1 to 2 command-line
arguments to work correctly and likely more if your goals are more than modest.
In the following sections, we present the configurations a new developer is most
likely to require.
In the following sections, we present the configurations a new developer is
most likely to require.

#### Selected Useful Testing Options
### Selected Useful Testing Options

We give an overview of common basic use cases here, and we provide full details
on the various capabilities and options of `test-all-eamxx` in
[EAMxx Automated Standalone Testing](dev_testing/test_all_eamxx.md).

!!! Warning "Note"

For all discussion of running standalone testing via `test-all-eamxx`,
including the following, we assume the commands are run from the root
directory of EAMxx. That is, if one begins in the root E3SM directory,
assume that each code snippet begins with an implicit

``` {.shell .copy}
$ eamxx_root="<some/file/path>/E3SM/components/eamxx"
$ cd "${eamxx_root}"
```

??? Example "Configure Only Without Build or Test"

**Useful for:** Mid-development testing when debugging compilation, and it
Expand Down Expand Up @@ -108,6 +120,7 @@ on the various capabilities and options of `test-all-eamxx` in
# will run tests of the same configuration using N threads \
--ctest-parallel-level <N> \
```

??? Quote "Only Run Requested Test Cases"

As a first step, you are likely to only be concerned with testing your
Expand Down Expand Up @@ -248,7 +261,7 @@ on the various capabilities and options of `test-all-eamxx` in
--make-parallel-level "${build_threads_gpu}"
```

#### Testing on Supported Machines
### Testing on Supported Machines

When testing on a supported E3SM machine, the only configuration required is to
set the proper "machine ID" flag that indicates the machine name and potentially
Expand Down Expand Up @@ -286,7 +299,7 @@ and efficiently[^mach-file].
$ ./scripts/test-all-eamxx --machine <machine-id>
```

#### Testing Locally
### Testing Locally

Running EAMxx on your local workstation, laptop, or less-trafficked cluster
machine can potentially make your development quicker and less complicated.
Expand All @@ -296,8 +309,8 @@ setup involved in making your life easier.
Testing on a local, or otherwise unknown, machine requires the `--local`
(or `-l`) flag and correctly-configured files in the `~/.cime/` directory.
For a full explanation of how to configure your own machine,
see [Local Configuration Files](dev_testing/test_all_eamxx.md#local-
configuration-files), but we will briefly summarize here.
see the pages on [Local Configuration Files](dev_testing/test_all_eamxx.md#local-
configuration-files) and [Third-party Libraries](TPLs.md), but we will briefly summarize here.

??? Example "Example `scream_mach_specs.py` Configuration File"

Expand Down Expand Up @@ -330,8 +343,6 @@ configuration-files), but we will briefly summarize here.
(mostly) more information is better than less.

```{.shell .copy title="${eamxx_root}/scripts/machine_specs.py:[L58-L88]"}
from machines_specs import Machine

###############################################################################
class Machine(object):
###############################################################################
Expand Down Expand Up @@ -376,6 +387,8 @@ configuration-files), but we will briefly summarize here.
concrete = True
@classmethod
def setup(cls):
# NOTE: do not change this line because `test-all-eamxx -l`
# specifically looks for a machine named "local".
super().setup_base("local")

# NOTE: these are run by the shell (likely bash), so must have
Expand Down Expand Up @@ -417,18 +430,12 @@ configuration-files), but we will briefly summarize here.
###############################################################################
```

<!-- ## To-do

- Mention full-model, but don't go into detail
- Discuss catch2 (add to comp. structures section)
- `create_unit_test()`
- Please use labels for tests -->

<!-- ======================================================================= -->

[^automagic-cmake]: In reality, significant effort is involved in designing a project to build "automagically," but the idea is to take that process out of the user's hands.
[^supported-machines]: If you're feeling brave and want to read some python, you can find details about supported machines and their configurations in [machine_specs.py](https://github.com/E3SM-Project/E3SM/blob/master/components/eamxx/scripts/machines_specs.py) in the `scripts/` directory. Or, for general machine files for E3SM, you can take a look at `E3SM/cime_config/machines/`
[^but-mac]: Mac computers with an Apple silicon CPU (e.g., M1, M2...) are not officially supported and present challenges related to handling floating-point exceptions. However, some have had success building on this architecture, and if you achieve a robust build configuration for Mac please reach out to the development team!
[^TPLs-etc]: In particular, it is likely you will need to manually build/install the required [Third-party libraries](TPLs.md).
[^smoke-test-def]: ***Smoke Test*** is a term used by software developers to describe a type of test that initially indicates whether something is working properly--as in, "flip the switch and see if anything starts smoking." :fingers_crossed:
<!-- doesn't like the separation between footnote and text reference appears
to be triggered by the admonition environment -->
Expand Down
3 changes: 2 additions & 1 deletion components/eamxx/mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ nav:
- 'Fields': 'developer/field.md'
- 'Grids and Remappers': 'developer/grid.md'
- 'Atmosphere Processes': 'developer/processes.md'
- 'I/O': 'developer/io.md'
- 'Input/Output': 'developer/io.md'
- 'Third-party Libraries': 'developer/TPLs.md'
- 'Technical Guide':
- 'Overview': 'technical/index.md'
- 'AeroCom cloud top': 'technical/aerocom_cldtop.md'
Expand Down
3 changes: 3 additions & 0 deletions mkdocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ markdown_extensions:
generic: true
- md_in_html
- tables
- pymdownx.emoji:
emoji_index: !!python/name:material.extensions.emoji.twemoji
emoji_generator: !!python/name:material.extensions.emoji.to_svg

plugins:
- monorepo
Expand Down