Small mods in test-all-scream related to baseline generation #3038

bartgol · 2024-10-15T19:48:32Z

Since generation will never happen along with running tests, use same folder that we use in test phase when generating.

This will allow certain scripts (e.g., gh actions that will come soon) to find build folders without checking whether this was a baselines generation run or not.

Edit: to expand on the motivation, in the gh actions I wand to upload test-all ctest logs as artifacts to the action (for debug/inspection). To specify the logs path, I am doing ctest-build/*/Testing/Temporary/*.log, but this is currently failing for baseline gen, since the correct path would be ctest-build/tas_baseline_build/*/Testing/Temporary/*.log.

Since generation will never happen along with running tests, use same folder that we use in test phase when generating. This will allow certain scripts to find build folders without checking whether this was a baselines generation run or not.

E3SM-Bot · 2024-10-15T20:15:43Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6159
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1e0148d`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5915
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1e0148d`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-10-15T21:29:26Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6159
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1e0148d`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5915
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1e0148d`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Weaver # 6159 PASSED (click to see last 100 lines of console output)


142/157 Test #142: model_initial .........................................................   Passed    5.80 sec
        Start 143: model_restart
143/157 Test #143: model_restart .........................................................   Passed    7.22 sec
        Start 144: restarted_vs_monolithic_check_np1
144/157 Test #144: restarted_vs_monolithic_check_np1 .....................................   Passed    0.10 sec
        Start 145: homme_shoc_cld_spa_p3_rrtmgp_np1
145/157 Test #145: homme_shoc_cld_spa_p3_rrtmgp_np1 ......................................   Passed    6.24 sec
        Start 146: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
146/157 Test #146: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp .............................   Passed    0.14 sec
        Start 147: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
147/157 Test #147: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ............................   Passed    8.82 sec
        Start 148: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
148/157 Test #148: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 .................   Passed    1.42 sec
        Start 149: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
149/157 Test #149: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp ...................   Passed    0.67 sec
        Start 150: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
150/157 Test #150: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 ...............................   Passed   19.22 sec
        Start 151: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
151/157 Test #151: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ......................   Passed    0.15 sec
        Start 152: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
152/157 Test #152: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 ...............................   Passed   18.17 sec
        Start 153: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
153/157 Test #153: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ......................   Passed    0.18 sec
        Start 154: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1
154/157 Test #154: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1 ............   Passed   18.50 sec
        Start 155: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
155/157 Test #155: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp ...   Passed    0.20 sec
        Start 156: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1
156/157 Test #156: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1 .........................   Passed   38.79 sec
        Start 157: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
157/157 Test #157: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp ................   Passed    0.15 sec
100% tests passed, 0 tests failed out of 157
Label Time Summary:

baseline_cmp               = 148.04 secproc (23 tests)

baseline_gen               = 326.53 secproc (25 tests)

bfbhash                    =   1.01 secproc (1 test)

check                      =   0.89 secproc (1 test)

cld                        =  51.95 secproc (7 tests)

cld_fraction               =   1.18 secproc (1 test)

cxx baseline_cmp           =   5.87 secproc (2 tests)

diagnostics                =  45.16 secproc (23 tests)

driver                     =  90.13 secproc (16 tests)

dynamics                   =   9.28 secproc (3 tests)

fail                       =  28.86 secproc (5 tests)

io                         =  58.59 secproc (14 tests)

mam4_aci                   =  29.70 secproc (4 tests)

mam4_constituent_fluxes    =   4.46 secproc (1 test)

mam4_drydep                =   3.52 secproc (1 test)

mam4_optics                =   4.31 secproc (1 test)

mam4_srf_online_emiss      =   4.46 secproc (1 test)

mam4_wetscav               =  17.30 secproc (2 tests)

nudging                    =  13.47 secproc (2 tests)

p3                         = 113.85 secproc (12 tests)

p3_sk                      =  31.06 secproc (2 tests)

physics                    = 180.03 secproc (27 tests)

remap                      =   5.92 secproc (1 test)

rrtmgp                     =  52.18 secproc (11 tests)

shoc                       =  63.42 secproc (13 tests)

spa                        =  12.41 secproc (4 tests)

surface_coupling           =   4.96 sec*proc (1 test)
Total Test time (real) = 800.15 sec
Testing '''1e0148d3cde0d38f768552ce690269bee3a69a80''' for test '''full_sp_debug'''
RUN: taskset -c 52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''1e0148d3cde0d38f768552ce690269bee3a69a80''' for test '''release'''
RUN: taskset -c 104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/release
Testing '''1e0148d3cde0d38f768552ce690269bee3a69a80''' for test '''full_debug'''
RUN: taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6159/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins5160311978689882925.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

SCREAM_PullRequest_Autotester_Mappy # 5915 FAILED (click to see last 100 lines of console output)


===============================================================================
RUN: cmake -C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5915/scream/components/eamxx/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-fortran/4.6.1/gcc/11.4.0/openmpi/4.1.6/rztypf2 -DNetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/netcdf-c/4.9.2/gcc/11.4.0/openmpi/4.1.6/po6gjlz -DPnetCDF_C_PATH=/projects/sems/install/rhel9-x86_64/sems/tpl/parallel-netcdf/1.12.3/gcc/11.4.0/openmpi/4.1.6/llgqd5t -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_ONLY_GENERATE_BASELINES=ON -DSCREAM_BASELINES_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5915/scream/components/eamxx/ctest-build/baselines/full_sp_debug -DSCREAM_TEST_MAX_TOTAL_THREADS=128 /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5915/scream/components/eamxx
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5915/scream/components/eamxx/ctest-build/full_sp_debug
WARNING: Failed to create baselines (config phase):
WARNING! Homme dycore cannot be used in a Single Precision build. Turning Homme off.
Error! Baselines tests enabled, but baseline dir is invalid.
  SCREAM_BASELINES_DIR: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5915/scream/components/eamxx/ctest-build/baselines/full_sp_debug
CMake Error at CMakeLists.txt:398 (message):
  Aborting...
OVERALL STATUS: FAIL
ERRPUT: 

Ran 33 tests in 405.804s
FAILED (failures=1, skipped=4)

Warning: repo is not clean: M foo

[[ 1 != 0 ]]
fails=0+1
scripts_fail=1
./scripts/cime-nml-tests

test_append (main.TestBuildnml)

Test that var+=value syntax behaves as expected ... ok

test_atmchanges_are_lost_with_hack_xml (main.TestBuildnml)

Test that atmchanges are lost if SCREAM_HACK_XML=TRUE ... ok

test_atmchanges_are_preserved (main.TestBuildnml)

Test that atmchanges are not lost when eamxx setup is called ... ok

test_atmchanges_are_preserved_testmod (main.TestBuildnml)

Test that atmchanges via testmod are preserved ... ok

test_atmchanges_for_atm_procs_add (main.TestBuildnml)

Test atmchanges that add atm procs ... ok

test_atmchanges_for_atm_procs_add_group (main.TestBuildnml)

Test atmchanges that add atm proc groups ... ok

test_atmchanges_for_atm_procs_add_invalid (main.TestBuildnml)

Test atmchanges that add atm procs ... ok

test_atmchanges_for_atm_procs_remove (main.TestBuildnml)

Test atmchanges that remove atm procs ... ok

test_atmchanges_on_all_matches (main.TestBuildnml)

Test that atmchange works with ANY ... ok

test_atmchanges_on_all_matches_plus_spec (main.TestBuildnml)

Test atmchange with ANY followed by an atmchange of one of them ... ok

test_atmchanges_on_arrays (main.TestBuildnml)

Test that atmchange works for array data ... ok

test_atmchanges_with_namespace (main.TestBuildnml)

Test that atmchange works when using 'namespace' syntax foo::bar ... ok

test_buffer_unchanged_with_bad_change_syntax (main.TestBuildnml)

Test atmchange does not change buffer if syntax was wrong ... ok

test_doctests (main.TestBuildnml)

Run doctests for all eamxx/cime_config python files and nml-related files in scripts ... ok

test_invalid_xml_option (main.TestBuildnml)

Test atmchange errors out with invalid param names ... ok

test_nml_defaults_append (main.TestBuildnml)

Test that the append attribute for array-type params in namelist defaults works as expected ... ok

test_pylint (main.TestBuildnml)

Run pylint on all eamxx/cime_config python files and nml-related files in scripts ... ok

test_reset_atmchanges_are_lost (main.TestBuildnml)

Test that atmchanges are lost when resetting ... ok

test_xmlchange_propagates_to_atmconfig (main.TestBuildnml)

Test that xmlchanges impact atm config files ... ok


Ran 19 tests in 231.612s
OK

[[ 0 != 0 ]]
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

######################################################

FAILS DETECTED:

SCREAM SCRIPTS TESTING FAILED!

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 1692077 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins1732518762633301654.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

Individual builds subdirs are already rm-ed

E3SM-Bot · 2024-10-16T00:15:12Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6162
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5918
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-10-16T01:05:10Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6162
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5918
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-10-16T01:05:27Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

E3SM-Bot · 2024-10-16T01:05:33Z

All Jobs Finished; status = PASSED, target_sha=75ef2edca2837f1f61601a83fba2b559b26df85f, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-10-16T21:11:06Z

The base branch has been updated since the last successful testing.

last PASS base branch sha: 75ef2ed
current base branch sha : 10fd3d0
The AutoTester will discard the last PASS, and re-test the PR from scratch

E3SM-Bot · 2024-10-16T21:14:01Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6172
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5924
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

bartgol · 2024-10-16T22:37:24Z

Jim previously approved, and tests pass now. Merging.

E3SM-Bot · 2024-10-16T22:37:45Z

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has closed the PR before testing completed. The Jenkins Jobs will be shutdown

E3SM-Bot · 2024-10-16T22:38:02Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6172
Status: ERROR

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5924
Status: ERROR

Jenkins Parameters

Parameter Name	Value
PR_LABELS	scripts;AT: Skip v1 Testing;code usability
PULLREQUESTNUM	3038
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`1161e21`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`75ef2ed`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Weaver # 6172 ERROR (click to see last 100 lines of console output)


142/157 Test #142: model_initial .........................................................   Passed    5.83 sec
        Start 143: model_restart
143/157 Test #143: model_restart .........................................................   Passed    7.11 sec
        Start 144: restarted_vs_monolithic_check_np1
144/157 Test #144: restarted_vs_monolithic_check_np1 .....................................   Passed    0.11 sec
        Start 145: homme_shoc_cld_spa_p3_rrtmgp_np1
145/157 Test #145: homme_shoc_cld_spa_p3_rrtmgp_np1 ......................................   Passed   10.52 sec
        Start 146: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
146/157 Test #146: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp .............................   Passed    0.14 sec
        Start 147: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
147/157 Test #147: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ............................   Passed    8.75 sec
        Start 148: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
148/157 Test #148: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 .................   Passed    1.50 sec
        Start 149: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
149/157 Test #149: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp ...................   Passed    0.64 sec
        Start 150: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
150/157 Test #150: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 ...............................   Passed   12.95 sec
        Start 151: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
151/157 Test #151: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ......................   Passed    0.13 sec
        Start 152: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
152/157 Test #152: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 ...............................   Passed   16.75 sec
        Start 153: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
153/157 Test #153: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ......................   Passed    0.18 sec
        Start 154: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1
154/157 Test #154: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_np1 ............   Passed   17.51 sec
        Start 155: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
155/157 Test #155: homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp ...   Passed    0.19 sec
        Start 156: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1
156/157 Test #156: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_np1 .........................   Passed   38.15 sec
        Start 157: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
157/157 Test #157: homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp ................   Passed    0.18 sec
100% tests passed, 0 tests failed out of 157
Label Time Summary:

baseline_cmp               = 124.01 secproc (23 tests)

baseline_gen               = 320.56 secproc (25 tests)

bfbhash                    =   0.93 secproc (1 test)

check                      =   0.94 secproc (1 test)

cld                        =  51.13 secproc (7 tests)

cld_fraction               =   1.20 secproc (1 test)

cxx baseline_cmp           =   5.65 secproc (2 tests)

diagnostics                =  49.10 secproc (23 tests)

driver                     = 105.29 secproc (16 tests)

dynamics                   =   7.42 secproc (3 tests)

fail                       =  27.99 secproc (5 tests)

io                         =  56.95 secproc (14 tests)

mam4_aci                   =  32.92 secproc (4 tests)

mam4_constituent_fluxes    =   7.98 secproc (1 test)

mam4_drydep                =   3.73 secproc (1 test)

mam4_optics                =   8.87 secproc (1 test)

mam4_srf_online_emiss      =   7.98 secproc (1 test)

mam4_wetscav               =  24.54 secproc (2 tests)

nudging                    =  19.08 secproc (2 tests)

p3                         = 120.42 secproc (12 tests)

p3_sk                      =  36.44 secproc (2 tests)

physics                    = 212.93 secproc (27 tests)

remap                      =   5.98 secproc (1 test)

rrtmgp                     =  54.52 secproc (11 tests)

shoc                       =  67.20 secproc (13 tests)

spa                        =  15.27 secproc (4 tests)

surface_coupling           =   1.77 sec*proc (1 test)
Total Test time (real) = 837.05 sec
Testing '''a4ddd2644da4ef94b153329b9fbb99612af56288''' for test '''full_sp_debug'''
RUN: taskset -c 52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''a4ddd2644da4ef94b153329b9fbb99612af56288''' for test '''release'''
RUN: taskset -c 104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/release
Testing '''a4ddd2644da4ef94b153329b9fbb99612af56288''' for test '''full_debug'''
RUN: taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.1/gcc/11.3.0/openmpi/4.1.6/5tv5psl -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.2/gcc/11.3.0/openmpi/4.1.6/pyuuqd3 -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.6/2s52shy -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6172/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins15922060285302169172.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

SCREAM_PullRequest_Autotester_Mappy # 5924 ERROR (click to see last 100 lines of console output)


+ [[ 1 == 1 ]]
+ ./scripts/scripts-tests -f -m mappy
test_doctests (__main__.TestGatherAllData) ... ok
test_full (__main__.TestGatherAllData) ... ok
test_pylint (__main__.TestGatherAllData) ... ok
test_doctests (__main__.TestMachinesSpecs) ... ok
test_full (__main__.TestMachinesSpecs) ... ok
test_pylint (__main__.TestMachinesSpecs) ... ok
test_doctests (__main__.TestScriptsTest) ... ok
test_full (__main__.TestScriptsTest) ... ok
test_pylint (__main__.TestScriptsTest) ... ok
test_at_test_level (__main__.TestTestAllScream)
Test that the at test level works ... ok
test_baseline_handling (__main__.TestTestAllScream)
Test the mechanics of test-all-scream's handling of baselines ... ok
test_build_fail_captured (__main__.TestTestAllScream)
Test that the 'dbg' test in test-all-scream detects and returns non-zero if there's a build error ... ok
test_config_fail_captured (__main__.TestTestAllScream)
Test that the 'dbg' test in test-all-scream detects and returns non-zero if there's a cmake configure error ... ok
test_dbg_details (__main__.TestTestAllScream)
Test the 'dbg' test in test-all-scream. It should set certain CMake values ... ok
test_doctests (__main__.TestTestAllScream) ... ok
test_experimental_test_level (__main__.TestTestAllScream)
Test that the experimental test level works ... ok
test_fpe_details (__main__.TestTestAllScream)
Test the 'fpe' test in test-all-scream. It should set certain CMake values ... ok
test_full (__main__.TestTestAllScream) ... ok
test_jenkins_process_at (__main__.TestTestAllScream)
Test the jenkins script in autotester mode, should return zero ... skipped 'Skipping jenkins test'
test_jenkins_process_fail_captured_at (__main__.TestTestAllScream)
Test the jenkins script in autotester mode with a failure, should return non-zero ... skipped 'Skipping jenkins test'
test_jenkins_process_fail_captured_nightly (__main__.TestTestAllScream)
Test the jenkins script in nightly mode with a failure, should return non-zero ... skipped 'Skipping jenkins test'
test_jenkins_process_nightly (__main__.TestTestAllScream)
Test the jenkins script in nightly mode, should return zero ... skipped 'Skipping jenkins test'
test_mem_check_details (__main__.TestTestAllScream)
Test the mem (default mem-check build) in test-all-scream. It should set certain CMake values ... ok
test_mpirank_and_thread_spreading (__main__.TestTestAllScream)
Test that MPI ranks and threads are properly spread across a machine's computing resources. ... ok
test_nightly_test_level (__main__.TestTestAllScream)
Test that the nightly test level works ... ok
test_opt_details (__main__.TestTestAllScream)
Test the 'opt' test in test-all-scream. It should set certain CMake values ... ok
test_pylint (__main__.TestTestAllScream) ... ok
test_run_fail_captured (__main__.TestTestAllScream)
Test that a failure from ctest in the testing phase is caught by test-all-scream ... ok
test_sp_details (__main__.TestTestAllScream)
Test the 'sp' test in test-all-scream. It should set certain CMake values ... ok
test_doctests (__main__.TestUtils) ... ok
test_full (__main__.TestUtils) ... ok
test_git_utils (__main__.TestUtils) ... ok
test_pylint (__main__.TestUtils) ... ok

Ran 33 tests in 318.713s
OK (skipped=4)

Warning: repo is not clean: M foo

[[ 0 != 0 ]]
./scripts/cime-nml-tests

test_append (main.TestBuildnml)

Test that var+=value syntax behaves as expected ... ok

test_atmchanges_are_lost_with_hack_xml (main.TestBuildnml)

Test that atmchanges are lost if SCREAM_HACK_XML=TRUE ... ok

test_atmchanges_are_preserved (main.TestBuildnml)

Test that atmchanges are not lost when eamxx setup is called ... $ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 2640426 killed;

[ssh-agent] Stopped.

Build was aborted

Aborted by Luca Bertagna

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins17496855939699131944.sh

Terminated

[[ 143 != 0 ]]
fails=0+1
scripts_fail=1
[[ -z '' ]]
[[ 1 == 1 ]]
[[ 0 == 1 ]]
[[ 0 == 1 ]]
echo 'SCREAM v1 tests were skipped, since the Github label '''AT: Skip v1 Testing''' was found.\n'

SCREAM v1 tests were skipped, since the Github label 'AT: Skip v1 Testing' was found.\n
set +x

######################################################

FAILS DETECTED:

SCREAM SCRIPTS TESTING FAILED!

######################################################

bartgol added scripts CI: skip eamxx-v1 Skip eamxx CIME testing for this PR code usability labels Oct 15, 2024

bartgol requested a review from jgfouca October 15, 2024 19:48

bartgol self-assigned this Oct 15, 2024

jgfouca previously approved these changes Oct 15, 2024

View reviewed changes

EAMxx: no need to rm work dir when generating baselines

1161e21

Individual builds subdirs are already rm-ed

bartgol dismissed jgfouca’s stale review via 1161e21 October 15, 2024 23:53

bartgol merged commit 41f563d into master Oct 16, 2024
3 of 5 checks passed

bartgol deleted the bartgol/eamxx/tas-minor-mod branch October 16, 2024 22:37

Small mods in test-all-scream related to baseline generation #3038

Small mods in test-all-scream related to baseline generation #3038

Uh oh!

Conversation

bartgol commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

E3SM-Bot commented Oct 15, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

Uh oh!

E3SM-Bot commented Oct 15, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Total Test time (real) = 800.15 sec

Testing '''1e0148d3cde0d38f768552ce690269bee3a69a80''' for test '''full_sp_debug'''

Testing '''1e0148d3cde0d38f768552ce690269bee3a69a80''' for test '''release'''

Testing '''1e0148d3cde0d38f768552ce690269bee3a69a80''' for test '''full_debug'''

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Uh oh!

E3SM-Bot commented Oct 16, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

Uh oh!

E3SM-Bot commented Oct 16, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Uh oh!

E3SM-Bot commented Oct 16, 2024

Uh oh!

E3SM-Bot commented Oct 16, 2024

Uh oh!

E3SM-Bot commented Oct 16, 2024

Uh oh!

E3SM-Bot commented Oct 16, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Using Repos:

Uh oh!

bartgol commented Oct 16, 2024

Uh oh!

Uh oh!

E3SM-Bot commented Oct 16, 2024

Uh oh!

E3SM-Bot commented Oct 16, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Total Test time (real) = 837.05 sec

Testing '''a4ddd2644da4ef94b153329b9fbb99612af56288''' for test '''full_sp_debug'''

Testing '''a4ddd2644da4ef94b153329b9fbb99612af56288''' for test '''release'''

Testing '''a4ddd2644da4ef94b153329b9fbb99612af56288''' for test '''full_debug'''

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Uh oh!

Uh oh!

bartgol commented Oct 15, 2024 •

edited

Loading