Skip to content

Add libfabric/1.15.2.0 module to frontier-scream-gpu. #2971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 27, 2024
Merged

Add libfabric/1.15.2.0 module to frontier-scream-gpu. #2971

merged 1 commit into from
Aug 27, 2024

Conversation

trey-ornl
Copy link
Contributor

The default version of module libfabric on Frontier changed from 1.15.2.0 to 1.20.1, and this caused hangs in the decadal run. I confirmed that the run hangs even if compiled with libfabric/1.20.1 when using the current cpe/22.12-based software environment, but it runs to completion with the module libfabric/1.15.2.0.

I also combined the two <module> blocks into one, since they both actually depend on the setting compiler="crayclang-scream".

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; this inspection will remain valid until a new commit to source branch is performed.

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5790
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6016
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (trey-ornl/scream)
  • Branch: trey/cime_config/frontier-libfabric
  • SHA: 8d324b2
  • Mode: TEST_REPO

Pull Request Author: trey-ornl

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5790
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6016
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5790 FAILED (click to see last 100 lines of console output) 

+ '[' -r /etc/profile.d/vte.sh ']'
+ '[' hxB '!=' hxB ']'
+ . /etc/profile.d/vte.sh
++ '[' -n '4.2.46(2)-release' -o -n '' ']'
++ [[ hxB == *i* ]]
++ return 0
+ for i in '/etc/profile.d/*.sh' /etc/profile.d/sh.local
+ '[' -r /etc/profile.d/which2.sh ']'
+ '[' hxB '!=' hxB ']'
+ . /etc/profile.d/which2.sh
++ alias 'which=alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
+ for i in '/etc/profile.d/*.sh' /etc/profile.d/sh.local
+ '[' -r /etc/profile.d/sh.local ']'
+ '[' hxB '!=' hxB ']'
+ . /etc/profile.d/sh.local
+ unset i
+ unset -f pathmunge
+ '[' -f /ascldap/users/e3sm-jenkins/.bashrc ']'
+ . /ascldap/users/e3sm-jenkins/.bashrc
++ '[' -f /etc/bashrc ']'
++ . /etc/bashrc
+++ '[' '' ']'
+++ shopt -q login_shell
+++ PATH=/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin
+ PATH=/usr/lib64/qt-3.3/bin:/usr/condabin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/opt/dell/srvadmin/sbin:/ascldap/users/e3sm-jenkins/.local/bin:/ascldap/users/e3sm-jenkins/bin
+ export PATH
+ cd /ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790
+ cd scream
+ ./components/eamxx/scripts/git-merge-ref origin/master
+ cd -
/ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790
+ ./scream/components/eamxx/scripts/jenkins/jenkins_common.sh
+++ dirname ./scream/components/eamxx/scripts/jenkins/jenkins_common.sh
++ cd ./scream/components/eamxx/scripts/jenkins
++ pwd
+ export JENKINS_SCRIPT_DIR=/ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790/scream/components/eamxx/scripts/jenkins
+ JENKINS_SCRIPT_DIR=/ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790/scream/components/eamxx/scripts/jenkins
++ date +%Y-%m-%d_%H%M%S
+ DATE_STAMP=2024-08-26_181646
+ set -o pipefail
+ /ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790/scream/components/eamxx/scripts/jenkins/jenkins_common_impl.sh
+ tee JENKINS_2024-08-26_181646
+ IFS=';'
+ read -r -a labels
+ skip_testing=0
+ test_scripts=0
+ test_v0=0
+ test_v1=1
+ test_SA=1
+ skip_mappy=0
+ skip_weaver=0
+ skip_blake=0
+ is_at_run=0
+ '[' 1 -gt 0 ']'
+ for label in '"${labels[@]}"'
+ '[' 'AT: AUTOMERGE' == 'AT: Integrate Without Testing' ']'
+ '[' 'AT: AUTOMERGE' == 'AT: Skip Stand-Alone Testing' ']'
+ '[' 'AT: AUTOMERGE' == 'AT: Skip v1 Testing' ']'
+ '[' 'AT: AUTOMERGE' == scripts ']'
+ '[' 'AT: AUTOMERGE' == SCREAMv0 ']'
+ '[' 'AT: AUTOMERGE' == 'AT: Skip mappy' ']'
+ '[' 'AT: AUTOMERGE' == 'AT: Skip weaver' ']'
+ '[' 'AT: AUTOMERGE' == 'AT: Skip blake' ']'
+ '[' 0 -eq 0 ']'
+ v0_fail=0
+ v1_fail=0
+ sa_fail=0
+ cov_fail=0
+ scripts_fail=0
+ memcheck_fail=0
+ cd /ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790/scream/components/eamxx/scripts/jenkins/../..
+ source scripts/jenkins/mappy_setup
++ module purge
+++ /usr/bin/modulecmd bash purge
++ eval
++ source /projects/sems/modulefiles/utils/sems-archive-modules-init.sh
/ascldap/users/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5790/scream/components/eamxx/scripts/jenkins/jenkins_common_impl.sh: line 2: /projects/sems/modulefiles/utils/sems-archive-modules-init.sh: No such file or directory
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 48014 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins11675878246401781142.sh

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 6016 PASSED (click to see last 100 lines of console output) 

136/152 Test #136: model_baseline ........................................................   Passed   12.46 sec
        Start 137: model_initial
137/152 Test #137: model_initial .........................................................   Passed    6.00 sec
        Start 138: model_restart
138/152 Test #138: model_restart .........................................................   Passed    6.98 sec
        Start 139: restarted_vs_monolithic_check_np1
139/152 Test #139: restarted_vs_monolithic_check_np1 .....................................   Passed    0.10 sec
        Start 140: homme_shoc_cld_spa_p3_rrtmgp_np1
140/152 Test #140: homme_shoc_cld_spa_p3_rrtmgp_np1 ......................................   Passed    6.29 sec
        Start 141: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
141/152 Test #141: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp .............................   Passed    0.12 sec
        Start 142: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
142/152 Test #142: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ............................   Passed   15.50 sec
        Start 143: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
143/152 Test #143: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 .................   Passed    1.45 sec
        Start 144: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
144/152 Test #144: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp ...................   Passed    0.61 sec
        Start 145: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
145/152 Test #145: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 ...............................   Passed   19.40 sec
        Start 146: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
146/152 Test #146: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ......................   Passed    0.08 sec
        Start 147: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
147/152 Test #147: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 ...............................   Passed   18.58 sec
        Start 148: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
148/152 Test #148: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ......................   Passed    0.21 sec
        Start 149: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1
149/152 Test #149: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1 ............   Passed   19.86 sec
        Start 150: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
150/152 Test #150: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp ...   Passed    0.15 sec
        Start 151: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1
151/152 Test #151: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1 .........................   Passed   38.98 sec
        Start 152: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
152/152 Test #152: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp ................   Passed    0.15 sec

100% tests passed, 0 tests failed out of 152


Label Time Summary:

baseline_cmp        = 118.23 secproc (21 tests)

baseline_gen        = 324.73 secproc (23 tests)

bfbhash             =   0.88 secproc (1 test)

check               =   0.90 secproc (1 test)

cld                 =  59.20 secproc (7 tests)

cld_fraction        =   4.47 secproc (1 test)

cxx baseline_cmp    =   7.67 secproc (2 tests)

diagnostics         =  54.89 secproc (23 tests)

driver              = 116.06 secproc (15 tests)

dynamics            =   5.62 secproc (3 tests)

fail                =  32.70 secproc (5 tests)

io                  =  56.08 secproc (14 tests)

mam4_aci            =  35.47 secproc (4 tests)

mam4_drydep         =   3.59 secproc (1 test)

mam4_optics         =  10.47 secproc (1 test)

mam4_wetscav        =  26.53 secproc (2 tests)

nudging             =  13.47 secproc (2 tests)

p3                  = 122.94 secproc (12 tests)

p3_sk               =  50.29 secproc (2 tests)

physics             = 221.79 secproc (26 tests)

remap               =   6.27 secproc (1 test)

rrtmgp              =  61.17 secproc (11 tests)

shoc                =  75.27 secproc (13 tests)

spa                 =  13.12 secproc (4 tests)

surface_coupling    =   5.28 sec*proc (1 test)


Total Test time (real) = 835.74 sec


Testing '''8d324b21cd3bb90b1f8d28c5631e1a1b02788d42''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''8d324b21cd3bb90b1f8d28c5631e1a1b02788d42''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/full_debug


Testing '''8d324b21cd3bb90b1f8d28c5631e1a1b02788d42''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx/ctest-build/release

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6016/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins17425212227535986148.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5791
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6017
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (trey-ornl/scream)
  • Branch: trey/cime_config/frontier-libfabric
  • SHA: 8d324b2
  • Mode: TEST_REPO

Pull Request Author: trey-ornl

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5791
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6017
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS AT: RETEST;AT: AUTOMERGE
PULLREQUESTNUM 2971
SCREAM_SOURCE_REPO https://github.com/trey-ornl/scream
SCREAM_SOURCE_SHA 8d324b2
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA de6c6d6
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5791 FAILED (click to see last 100 lines of console output) 

PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240827_144815_3sd3ns
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240827_144815_3sd3ns
test-scheduler took 1354.6537735462189 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240827_144815_3sd3ns
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240827_144815_3sd3ns
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240827_144815_3sd3ns
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu.C.20240827_144815_3sd3ns
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240827_144815_3sd3ns
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240827_144815_3sd3ns
test-scheduler took 1354.6537735462189 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
340:cosp_standalone_np1
341:cosp_standalone_np2
342:cosp_standalone_np3
343:cosp_standalone_np4
344:cosp_standalone_baseline_cmp
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240827_144815_3sd3ns
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240827_144815_3sd3ns
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240827_144815_3sd3ns
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01.C.20240827_144815_3sd3ns
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu.C.20240827_144815_3sd3ns
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240827_144815_3sd3ns
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240827_144815_3sd3ns
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240827_144815_3sd3ns
test-scheduler took 1354.6537735462189 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 28261 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins8471406453357828619.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 6017 PASSED (click to see last 100 lines of console output) 

136/152 Test #136: model_baseline ........................................................   Passed   11.71 sec
        Start 137: model_initial
137/152 Test #137: model_initial .........................................................   Passed    5.94 sec
        Start 138: model_restart
138/152 Test #138: model_restart .........................................................   Passed    7.29 sec
        Start 139: restarted_vs_monolithic_check_np1
139/152 Test #139: restarted_vs_monolithic_check_np1 .....................................   Passed    0.12 sec
        Start 140: homme_shoc_cld_spa_p3_rrtmgp_np1
140/152 Test #140: homme_shoc_cld_spa_p3_rrtmgp_np1 ......................................   Passed   12.07 sec
        Start 141: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
141/152 Test #141: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp .............................   Passed    0.12 sec
        Start 142: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
142/152 Test #142: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ............................   Passed   14.37 sec
        Start 143: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
143/152 Test #143: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 .................   Passed    1.53 sec
        Start 144: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
144/152 Test #144: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp ...................   Passed    0.64 sec
        Start 145: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
145/152 Test #145: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 ...............................   Passed   18.91 sec
        Start 146: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
146/152 Test #146: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ......................   Passed    0.10 sec
        Start 147: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
147/152 Test #147: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 ...............................   Passed   17.95 sec
        Start 148: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
148/152 Test #148: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ......................   Passed    0.16 sec
        Start 149: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1
149/152 Test #149: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1 ............   Passed   18.57 sec
        Start 150: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
150/152 Test #150: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp ...   Passed    0.17 sec
        Start 151: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1
151/152 Test #151: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1 .........................   Passed   38.98 sec
        Start 152: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
152/152 Test #152: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp ................   Passed    0.19 sec

100% tests passed, 0 tests failed out of 152


Label Time Summary:

baseline_cmp        = 148.37 secproc (21 tests)

baseline_gen        = 350.87 secproc (23 tests)

bfbhash             =   0.90 secproc (1 test)

check               =   0.91 secproc (1 test)

cld                 =  49.39 secproc (7 tests)

cld_fraction        =   1.21 secproc (1 test)

cxx baseline_cmp    =  10.99 secproc (2 tests)

diagnostics         =  58.21 secproc (23 tests)

driver              =  82.42 secproc (15 tests)

dynamics            =   7.84 secproc (3 tests)

fail                =  30.50 secproc (5 tests)

io                  =  62.30 secproc (14 tests)

mam4_aci            =  30.79 secproc (4 tests)

mam4_drydep         =   3.63 secproc (1 test)

mam4_optics         =   4.65 secproc (1 test)

mam4_wetscav        =  17.37 secproc (2 tests)

nudging             =   7.82 secproc (2 tests)

p3                  = 104.90 secproc (12 tests)

p3_sk               =  39.08 secproc (2 tests)

physics             = 172.87 secproc (26 tests)

remap               =   6.10 secproc (1 test)

rrtmgp              =  49.29 secproc (11 tests)

shoc                =  61.19 secproc (13 tests)

spa                 =   8.27 secproc (4 tests)

surface_coupling    =   5.25 sec*proc (1 test)


Total Test time (real) = 835.48 sec


Testing '''8d324b21cd3bb90b1f8d28c5631e1a1b02788d42''' for test '''full_sp_debug'''


RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/full_sp_debug


Testing '''8d324b21cd3bb90b1f8d28c5631e1a1b02788d42''' for test '''release'''


RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/release


Testing '''8d324b21cd3bb90b1f8d28c5631e1a1b02788d42''' for test '''full_debug'''


RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6017/scream/components/eamxx

Completed analysis on weaver'


    

  • [[ 0 != 0 ]]
    • 

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins7081350956742564342.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@ambrad
Copy link
Member

ambrad commented Aug 27, 2024

This is fine to merge with the current AT results.

@ambrad ambrad merged commit f8cdc92 into E3SM-Project:master Aug 27, 2024
5 of 6 checks passed
@trey-ornl trey-ornl deleted the trey/cime_config/frontier-libfabric branch August 27, 2024 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ambrad ambrad ambrad approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@trey-ornl @E3SM-Bot @ambrad