AMG2023 CUDA: doesn't build with Spack 1.0 due to linker errors with cublas and blas

[pearce8]

Same issue on Matrix as on Lassen. Please fix on Matrix.

[rfhaque]

@scheibelp @pearce8 Getting the code to correctly build on CUDA machines (matrix and lassen) required explicitly specifying the blas/lapack libraries in package.py

def setup_build_environment(self, env):
  if "+cuda" in self.spec:
    #env.append_flags('LDFLAGS', '-lessl') # lassen
    #env.append_flags('LDFLAGS', '-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lm') # matrix
    ...

This is a problem since the exact list of library names to be added can vary for the particular blas/lapack implementation used on the target system.
Also, this error does not happen on other machines such as tuolumne or matrix dane.

To reproduce:
Use branch amg_spack1_changes
bin/benchpark system init --dest=matrix llnl-matrix compiler=gcc cuda=12.6.0
bin/benchpark experiment init --dest amg2023 --system=matrix amg2023 +cuda~openmp caliper=cuda,mpi

[pearce8]

@rfhaque Please confirm whether this is an issue on matrix.

[scheibelp]

I can reproduce this now (as of a few minutes ago) on matrix

Side note: amg2023's git attribute must be replaced with "https://github.com/rfhaque/AMG2023.git" to reproduce this at the moment (Riyaz also mentioned that this should be reproducible if we set the amg2023 version to @develop)

[scheibelp]

I'm able to get a successful build on matrix if I apply the following diff to the amg2023 source:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6fdc5db..4d0014f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -47,6 +47,11 @@ include_directories(${HYPRE_INCLUDE_DIR})
 list(APPEND AMG_DEPENDENCIES
   ${HYPRE_LIBRARIES})
 
+find_package(BLAS   REQUIRED)
+find_package(LAPACK REQUIRED)
+
+list(APPEND AMG_DEPENDENCIES LAPACK::LAPACK BLAS::BLAS)
+
 if (AMG_WITH_CUDA)
   find_package(CUDAToolkit REQUIRED)
   list(APPEND AMG_DEPENDENCIES

I think amg2023 should in fact be doing that to link the blas libs it needs in general (though I didn't test this on other systems and/or with other blas/lapack implementations).

Another approach is to condition setup_build_environment on finding mkl in the spec:

def setup_build_environment(self, env):
  if "mkl" in self.spec:
    env.append_flags('LDFLAGS', '-lmkl_intel_lp64 -lmkl_sequential -lmkl_core')

[pearce8]

@rfhaque please take a look at @scheibelp's comments, and lets discuss on Monday.