Skip to content

Change output specs for mam tests #3066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 28, 2024
Merged

Change output specs for mam tests #3066

merged 2 commits into from
Oct 28, 2024

Conversation

bartgol
Copy link
Contributor

@bartgol bartgol commented Oct 25, 2024

The baselines tests only check the first file. So put all snapshots in the same file, not just the t0 snap, to make sure we're comparing the final state, not just the IC.

@bartgol bartgol added cmake Concerns CMake related files or logic AT: AUTOMERGE bugfix labels Oct 25, 2024
@bartgol bartgol self-assigned this Oct 25, 2024
@bartgol bartgol force-pushed the bartgol/eamxx/fix-mam-standalone-baselines-test branch from 89c86f7 to ed910a3 Compare October 25, 2024 21:32
@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6234
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA ed910a3
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 6c48c85
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5974
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA ed910a3
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 6c48c85
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/fix-mam-standalone-baselines-test
  • SHA: ed910a3
  • Mode: TEST_REPO

Pull Request Author: bartgol

@bartgol
EAMxx: change output specs for mam tests
88b1da8 
The baselines tests only check the first file, so put all
snapshots in the same file, not just the t0 snap
@bartgol
EAMxx: don't skip layer when using add_subdirectory
30b87a9 
It prevents us from running "make" or "ctest"
in the intermediate layer folder
@bartgol bartgol force-pushed the bartgol/eamxx/fix-mam-standalone-baselines-test branch from ed910a3 to 30b87a9 Compare October 25, 2024 22:08
@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKDMwYjg3YTkwYTA2NDM2ZGZjYTM4YjIzOWViY2ZkMDhmZjc4YWY0OGY... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6234
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA ed910a3
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 6c48c85
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5974
  • Status: ERROR

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA ed910a3
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 6c48c85
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Weaver # 6234 ERROR (click to see last 100 lines of console output) 

 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/eam/src/physics/crm/pam/external.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-seaice/src/icepack.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-seaice/src/icepack.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/haero.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/haero.path # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/mam4xx.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/mam4xx.path # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/fates # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/mpp # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/sbetr # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/eam/src/physics/rrtmgp/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/scorpio # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/ekat # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/YAKL # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/cub # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/cvmix # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/BGC # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/MARBL # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/gotm # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/mct # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-albany-landice/src/SeaLevelModel # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
Setting http proxy: proxy.sandia.gov:80
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/SHTNS # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
Setting http proxy: proxy.sandia.gov:80
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/FFTW # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
Setting http proxy: proxy.sandia.gov:80
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-ocean/src/ppr # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/ww3/src/WW3 # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/eam/src/physics/crm/pam/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/mpas-seaice/src/icepack # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive externals/haero # timeout=10
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins18193476905615620552.sh
SCREAM_PullRequest_Autotester_Mappy # 5974 ERROR (click to see last 100 lines of console output) 

............Build project
   Each symbol represents 1024 bytes of output.
   '\''!'\'' represents an error and '\''*'\'' a warning.
    ***.*.....*.***.*********.....****..********.*.****.**.**...*.**....**.*......................  Size: 99K
    ....  Size: 50K
    ................................................  Size: 49K
    ................................................................  Size: 99K
    ................................................  Size: 100K
    ..........................................................................................................  Size: 149K
    ...  Size: 150K
    ......................................................  Size: 149K
    ..............  Size: 199K
    .........................................................................  Size: 199K
    ....................................................  Size: 250K
    ......................................................  Size: 300K
    ................................................*........  Size: 349K
    ........................................................  Size: 399K
    ...................................................  Size: 450K
    ...................................................  Size: 499K
    ....................................................  Size: 549K
    ..................................................  Size: 599K
    ..................................................  Size: 649K
    ...................................................  Size: 700K
    ..................................................  Size: 749K
    ..................................................  Size: 799K
    ....................................................  Size: 849K
    ..................................................  Size: 899K
    ..................................................  Size: 950K
    ...................................................  Size: 1000K
    ....................................................  Size: 1050K
    ...................................................  Size: 1099K
    ..............................................  Size: 199K
    .....................  Size: 1150K
    ...............................................................  Size: 1200K
    ......................................................................  Size: 250K
    ...*.*.*.............*.*...................  Size: 1249K
    ................................  Size: 299K
    ..................................................  Size: 349K
    ..................................................  Size: 399K
    ..................................................  Size: 449K
    ..................................................  Size: 500K
    ...................................................  Size: 549K
    ..................................................  Size: 600K
    ..................................................  Size: 650K
    ..................................................  Size: 699K
    ..................................................  Size: 749K
    ..................................................  Size: 800K
    ...................................................  Size: 849K
    ..................................................  Size: 900K
    ..................................................  Size: 949K
    ..................................................  Size: 999K
    .....................................................  Size: 1049K
    ...................................................  Size: 1099K
    ..................................................  Size: 1149K
    ...................................................  Size: 1200K
    ..................................................  Size: 1249K
    .............................................*................................  Size: 249K
    ....................................................  Size: 299K
    ..................................................  Size: 350K
    .............................  Size: 49K
    .........................  Size: 399K
    ...................................................  Size: 449K
    ......................................................  Size: 499K
    .......*..*.......*...............................*.........  Size: 549K
    ....................................................  Size: 600K
    ....................................................  Size: 650K
    ..................................................  Size: 699K
    ....................................................  Size: 749K
    ..................................................  Size: 800K
    ..................................................  Size: 849K
    .....................................................  Size: 900K
    ..................................................  Size: 949K
    ....................................................  Size: 1000K
    ............................................................  Size: 1049K
    ........................................................  Size: 1100K
    ...................................................  Size: 1150K
    .......................................................  Size: 1200K
    ........  Size: 1299K
    ..............................................................  Size: 1249K
    ..........................  Size: 1299K
    .............................  Size: 1300K
    ......................................................  Size: 1350K
    .............................................................................  Size: 1399K
    .....***.**.....!.!.!!!..!..!!!!!....!Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5974/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5974/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5974/scream/components/eamxx
mappy failed
!!!!!.!!!.  Size: 100K
    .!!!!!!!!!!..!!!!!!!!!'
+ errors=
+ SA_FAILURES_DETAILS+=
+ [[ 1 == 0 ]]
+ [[ mappy == \m\a\p\p\y ]]
+ [[ 0 == 1 ]]
+ [[ -z '' ]]
+ [[ 1 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 1 == 1 ]]
++ ../../cime/scripts/create_test e3sm_scream_v1_at -c -b master --wait
Terminated

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6235
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 30b87a9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 0bf8e42
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5975
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 30b87a9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 0bf8e42
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/fix-mam-standalone-baselines-test
  • SHA: 30b87a9
  • Mode: TEST_REPO

Pull Request Author: bartgol

@bartgol
Copy link
Contributor Author

bartgol commented Oct 25, 2024

The eamxx-sa gh action checks that fail are expected, since we are adding more snapshots to the nc files saved by those three tests, hence the diff.

singhbalwinder
singhbalwinder approved these changes Oct 25, 2024
View reviewed changes
Copy link
Contributor

@singhbalwinder singhbalwinder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Luca for fixing this!

@E3SM-Bot
Copy link
Collaborator

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 6235
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 30b87a9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 0bf8e42
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5975
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS cmake;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 3066
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 30b87a9
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 0bf8e42
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Weaver # 6235 FAILED (click to see last 100 lines of console output) 


Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_drydep_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_drydep_standalone_baseline_cmp

152:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

154:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp

156:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx

weaver failed'


    

  • errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:
    
110:mam4_drydep_standalone_baseline_cmp
    
153:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
    
155:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
    
157:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_drydep_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_drydep_standalone_baseline_cmp

152:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

154:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp

156:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx

weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
    
110:mam4_drydep_standalone_baseline_cmp
    
153:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
    
155:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
    
157:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:

95:mam4_drydep_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:

109:mam4_drydep_standalone_baseline_cmp

152:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

154:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp

156:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx

weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
    
######################################################
    
FAILS DETECTED:
    
SCREAM STANDALONE TESTING FAILED!
    
Build type full_debug failed at testing time. Here's a list of failed tests:
    
110:mam4_drydep_standalone_baseline_cmp
    
153:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
    
155:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
    
157:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here's a list of failed tests:

95:mam4_drydep_standalone_baseline_cmp


Build type release failed at testing time. Here's a list of failed tests:

109:mam4_drydep_standalone_baseline_cmp

152:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

154:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp

156:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/6235/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins4206099811940511004.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
SCREAM_PullRequest_Autotester_Mappy # 5975 FAILED (click to see last 100 lines of console output) 

Starting RUN for test ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 in 3.304084 seconds (PEND). [COMPLETED 13 of 17]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 478.800348 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 1.313340 seconds (PEND). [COMPLETED 14 of 17]
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble in 520.179433 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble in 2.536264 seconds (PEND). [COMPLETED 15 of 17]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 463.449784 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 0.985580 seconds (PEND). [COMPLETED 16 of 17]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 1040.036554 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 0.595595 seconds (PEND). [COMPLETED 17 of 17]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20241025_173048_m7dqd1
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20241025_173048_m7dqd1
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20241025_173048_m7dqd1
PASS ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels--scream-output-preset-5 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels--scream-output-preset-5.C.20241025_173048_m7dqd1
PASS ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_p3--scream-output-preset-5 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_p3--scream-output-preset-5.C.20241025_173048_m7dqd1
PASS ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_shoc--scream-output-preset-5 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-small_kernels_shoc--scream-output-preset-5.C.20241025_173048_m7dqd1
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20241025_173048_m7dqd1
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97.C.20241025_173048_m7dqd1
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble.C.20241025_173048_m7dqd1
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01.C.20241025_173048_m7dqd1
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu.C.20241025_173048_m7dqd1
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20241025_173048_m7dqd1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20241025_173048_m7dqd1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-drydep.C.20241025_173048_m7dqd1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20241025_173048_m7dqd1
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20241025_173048_m7dqd1
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /home/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20241025_173048_m7dqd1
test-scheduler took 2072.1108844280243 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
368:mam4_drydep_standalone_baseline_cmp
534:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
542:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp
550:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp

Build type full_sp_debug failed at testing time. Here's a list of failed tests:

317:mam4_drydep_standalone_baseline_cmp


Build type release failed at testing time. Here's a list of failed tests:

367:mam4_drydep_standalone_baseline_cmp

533:homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp

541:homme_shoc_cld_mam_aci_p3_mam_optics_rrtmgp_mam_drydep_baseline_cmp

549:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5975/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5975/scream/components/eamxx && source /projects/sems/modulefiles/utils/sems-modules-init.sh && module purge && module load sems-cmake/3.27.9 sems-git/2.42.0 sems-gcc/11.4.0 sems-openmpi-no-cuda/4.1.6 sems-netcdf-c/4.9.2 sems-netcdf-cxx/4.2 sems-netcdf-fortran/4.6.1 sems-parallel-netcdf/1.12.3 sems-openblas && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5975/scream/components/eamxx

mappy failed

######################################################

Build step 'Execute shell' marked build as failure

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 807511 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins5603540691386277271.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

@singhbalwinder
Copy link
Contributor

I am not sure why mam4_drydep_standalone_baseline_cmp test is failing. The changes you introduced are all for multi-process tests, right? Do you know why this test is failing?

@bartgol
Copy link
Contributor Author

bartgol commented Oct 28, 2024

Failures are expected: one test wasn't blessed (but this PR does not touch it) and three baseline cmp were added. Merging

@bartgol bartgol merged commit 65dca4e into master Oct 28, 2024
8 of 11 checks passed
@bartgol bartgol deleted the bartgol/eamxx/fix-mam-standalone-baselines-test branch October 28, 2024 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@singhbalwinder singhbalwinder singhbalwinder approved these changes

Assignees

@bartgol bartgol

Labels
bugfix cmake Concerns CMake related files or logic
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bartgol @E3SM-Bot @singhbalwinder