Add build tools

[Zeitsperre]

Overview

This PR fixes #156 by adding relevant build packages to the Conda environment for compilation of C, C++, and Fortran libraries.

Changes

• Adds:
  • make (build runner for *nix systems)
  • cmake (cross-platform build runner, specific for raven-hydro)
  • c-compiler (metapackage for C)
  • cxx-compiler (metapackage for C++)
  • fortran-compiler (metapackage for Fortran)

Testing Checklist

• Committed the Jenkins build log to this Pull Request: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/jenkins-buildlogs-default.txt

Related Issue / Discussion

This PR does not require an immediate release. All this does is add the build-essential libraries so that users building and running C/C++/Fortran libraries no longer need to install those compilers manually in their Conda environments.

• Matching notebook fixes: N/A

• Deployment to PAVICS: PR URL

• Jenkins-config changes for new notebooks: N/A

• Previous release: N/A

Additional Information

Full diff of the conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/compare/release-py311-250423-update250730...NEW-release-py###-######

Full new conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/conda-env-export.yml

DockerHub build log:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/NEW-release-py###-######/docker/saved_buildout/docker-buildlogs.txt

[tlvu]

LGTM but can you make sure we can compile CLASSIC as in branch add-fortran-compiler-for-classic master...add-fortran-compiler-for-classic?

Then this PR can replace my branch above.

Don't forget to commit all the extra output files under docker/saved_buildout/ for tracking purposes.

[Zeitsperre]

@tlvu Sounds good.

Can you verify how you install CLASSIC? When checking the website, I get the impression that we need to install Singularity as well (there happens to be a Conda package, but's very old)?

If we just need a Fortran compiler, this PR already covers it.

[tlvu]

@tlvu Sounds good.

Can you verify how you install CLASSIC? When checking the website, I get the impression that we need to install Singularity as well (there happens to be a Conda package, but's very old)?

If we just need a Fortran compiler, this PR already covers it.

Did you look at my branch?

PAVICS-e2e-workflow-tests/docker/Dockerfile.testing

Lines 10 to 14 in 2d8e004

	#$ git clone https://gitlab.com/cccma/classic.git
	#$ cd classic
	#$ make
	#$ bin/CLASSIC_serial
	##shows help message

[Zeitsperre]

@tlvu It seems like this approach won't work for CLASSIC:

Despite libnetcdf-fortran being installed (f90==True), the CLASSIC code isn't set up to look for it?

Update: It works if we explicitly set the NetCDF-Fortran location in the Makefile. I think we can go forward with this as is.

FFLAGS = $(FFLAGS) -I/opt/conda/envs/birdy/include

[tlvu]

Update: It works if we explicitly set the NetCDF-Fortran location in the Makefile. I think we can go forward with this as is.

FFLAGS = $(FFLAGS) -I/opt/conda/envs/birdy/include

So the Makefile has to be modified? Or we can simply set this when invoking make so we can simply document this?

[Zeitsperre]

The Makefile needs to be modified, from what I can tell. For the type of user that would make use of this (advanced Fortran developers), I don't even think this constitutes a problem. It's possible that there is an option to invoke that configuration, but that's not entirely clear to me.

[Zeitsperre]

@tlvu Do we need to document this? Outside two Ouranos employees I know of that use CLASSIC, who else would this be supporting?

I'd be more inclined to modify the Makefile for CLASSIC on their repository to add some configurations to support conda instead.

[tlvu]

It's possible that there is an option to invoke

Trying my luck: make FFLAGS=-I/opt/conda/envs/birdy/include (setting make variable on the fly)? Or FFLAGS=-I/opt/conda/envs/birdy/include make (make it an env var when invoking make)?

[tlvu]

@tlvu Do we need to document this? Outside two Ouranos employees I know of that use CLASSIC, who else would this be supporting?

I'd be more inclined to modify the Makefile for CLASSIC on their repository to add some configurations to support conda instead.

Question for @huard I guess, he asked for this so he would know the use-case.

[Zeitsperre]

@tlvu The FFLAGS are supplied by the Makefile directly, no option to pass additional FFLAGS (that use-case wasn't thought of at the time, I suppose):

ifeq ($(mode), supercomputer)
	# Wrapper to the default Fortran compiler loaded via a module. The following is specific to the Intel compiler.
	# Include/library flags should NOT be specified as the appropriate ones should be available via the loaded module.
	COMPILER = ftn
	# Fortran Flags.
        FFLAGS = -DPARALLEL -r8 -g -O2 -mp1 -xCORE-AVX2 -align array64byte -init=arrays -init=zero -traceback -module $(ODIR)
else ifeq ($(mode), cray)
	# Wrapper to the default Fortran compiler loaded via a module. The following is specific to the Cray compiler.
	# Include/library flags should NOT be specified as the appropriate ones should be available via the loaded module.
	COMPILER = ftn
	# Fortran Flags. Note: -g prevents most optimizations, -rm gives listings.
	FFLAGS = -DPARALLEL -s real64 -e0 -ez -O2 -g
else ifeq ($(mode), ppp)
        # NB: Support for parallel netCDF library must first be enabled prior to running make via:
        #     . ssmuse-sh -x hpco/exp/hdf5-netcdf4/parallel/openmpi-3.1.2/static/intel-19.0.3.199/01
        # Open MPI wrapper to the default Fortran compiler. The following is specific to the Intel compiler.
        # Include/library flags for Open MPI should NOT be specified as the appropriate ones should be available via the wrapper.
        COMPILER = mpif90
        # Fortran Flags.
        FFLAGS = -DPARALLEL -r8 -g -O2 -mp1 -xCORE-AVX2 -align array64byte -init=arrays -init=zero -traceback -module $(ODIR)
        # For debugging:
        #FFLAGS = -DPARALLEL -r8 -g -O1 -mp1 -xCORE-AVX2 -align array64byte -init=arrays -init=zero -traceback -fpe0 -module $(ODIR)
        # Library Flags for netCDF.
        LFLAGS = -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz -lcurl
else ifeq ($(mode), parallel)
        # Parallel compiler.
        COMPILER = mpif90
        # Fortran Flags. The following is specific to the gfortran compiler.
        FFLAGS = -DPARALLEL -O3 -g -fdefault-real-8 -ffree-line-length-none -fbacktrace -ffpe-trap=invalid,zero,overflow -fbounds-check -J$(ODIR)
        # Include Flags.
        IFLAGS =  -I/para_netcdf_hdf-4.6.3/MPI/include
        # Library Flags.
        LFLAGS = -L/para_netcdf_hdf-4.6.3/MPI/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5
else
	# Serial compiler.
	COMPILER = gfortran
	mode = serial
	ODIR = objectFiles
	# Fortran Flags.
	FFLAGS = -O3 -g -fdefault-real-8 -ffree-line-length-none -fbacktrace -ffpe-trap=invalid,zero,overflow -fbounds-check -J$(ODIR) #-Wall -Wextra
	# Include Flags.
	IFLAGS = -I/usr/include
	# Library Flags
	LFLAGS = -lnetcdff -ldl -lz -lm
endif

### ADDED BY @Zeitsperre 
FFLAGS = $(FFLAGS) -I/opt/conda/envs/birdy/include

# Create required directory/.gitignore file, if missing.
VOID := $(shell mkdir -p $(ODIR))
#VOID := $(shell [ ! -f $(ODIR)/.gitignore ] && cp objectFiles/.gitignore $(ODIR))

# RECIPES
# Compile object files from .F90 sources
$(ODIR)/%.o: src/%.F90
	$(COMPILER) $(FFLAGS) -c $< -o $@

# Compile object files from .f90 sources
$(ODIR)/%.o: src/%.f90
	$(COMPILER) $(FFLAGS) $(IFLAGS) -c $< -o $@

# Compile object files from .f (Fortran 77) sources
$(ODIR)/%.o: src/%.f
	$(COMPILER) $(FFLAGS) $(IFLAGS) -c $< -o $@

# Properly reference the ODIR for the linking
OBJD = $(patsubst %,$(ODIR)/%,$(OBJ))

# Link objects together and put executable in the bin/ directory
CLASSIC: $(OBJD)
	$(COMPILER) $(FFLAGS) $(IFLAGS) -o bin/CLASSIC_$(mode) $(OBJD) $(LFLAGS)

# "make clean mode=supercomputer" removes all object files in objectFiles_supercomputer
clean:
	rm -f $(ODIR)/*.o $(ODIR)/*.mod bin/CLASSIC_$(mode)

Thankfully, there is a way to set ENV_VARS using conda, so we could technically have something set for birdy on activation (see: https://stackoverflow.com/a/62508395). But, just to be clear, we would still need to change the Makefile to accept this.

[tlvu]

LGTM, as long as @huard is OK with the extra work-around when compiling CLASSIC.

No need to generate output for docker/saved_buildout/, we can do next time we release a full build.

[Zeitsperre]

@tlvu Sounds good, I was just about to write the same thing.

The CLASSIC build workflow is several years without changes. I think it's fair to ask ECCC for (or provide ECCC with) a pull request to update their Makefile. Adding something as simple as this to the Makefile might work fine:

IFLAGS := $(CONDA_PREFIX)/include

[huard]

When I read https://docs.conda.io/projects/conda-build/en/stable/resources/compiler-tools.html#using-the-compiler-packages
I get the impression that there is way to alias the conda compilers with their standard name. But when I try this on the miniconda docker image, I can't get it to work. I mean, if this worked, I feel like it would be the ideal solution. You activate the environment, and then the usual commands simply point to the conda compilers.

[Zeitsperre]

@tlogan2000 PVI

[tlogan2000]

@tlogan2000 PVI

Thanks.
Just a word of caution that I think this can probably be put in place but could eventually be flagged as too big a security risk. We are hoping to have some external support for a security audit of our PAVICS infra sometime in the next year or so. Basically I am trying to say that if this becomes available now there is no guarantee that it will always be available on the platform