EAMxx: add support for ghci-snl-cuda in standalone testing #3077

bartgol · 2024-10-30T23:52:30Z

The AT2 support for this machine is still in its testing phase, but I need a PR to test it.

E3SM-Bot · 2024-10-31T00:13:32Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6256
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: workflow change approved
PULLREQUESTNUM	3077
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`6d3abef`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`d487f27`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-10-31T01:01:02Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6256
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: workflow change approved
PULLREQUESTNUM	3077
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`6d3abef`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`d487f27`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-10-31T01:01:18Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

E3SM-Bot · 2024-10-31T01:01:25Z

All Jobs Finished; status = PASSED, target_sha=d487f2744e35796e752c85a34eeb6a528591d117, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-10-31T01:15:47Z

The base branch has been updated since the last successful testing.

last PASS base branch sha: d487f27
current base branch sha : cb30379
The AutoTester will discard the last PASS, and re-test the PR from scratch

E3SM-Bot · 2024-10-31T02:51:48Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6257
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: workflow change approved
PULLREQUESTNUM	3077
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7aff00f`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`d487f27`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-10-31T03:39:15Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6257
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: workflow change approved
PULLREQUESTNUM	3077
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7aff00f`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`d487f27`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-10-31T03:39:31Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

E3SM-Bot · 2024-10-31T03:39:38Z

All Jobs Finished; status = PASSED, target_sha=cb30379332fb0a59dee9ab7105f6523f3575351e, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-10-31T06:14:30Z

The base branch has been updated since the last successful testing.

last PASS base branch sha: cb30379
current base branch sha : 3146f7f
The AutoTester will discard the last PASS, and re-test the PR from scratch

E3SM-Bot · 2024-10-31T06:15:12Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6262
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: workflow change approved
PULLREQUESTNUM	3077
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7aff00f`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`d487f27`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: bartgol

E3SM-Bot · 2024-10-31T07:01:48Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 6262
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: workflow change approved
PULLREQUESTNUM	3077
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`7aff00f`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`d487f27`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-10-31T07:02:05Z

All Jobs Finished; status = PASSED, target_sha=3146f7ff975b9364d30a63fc8de9e450e20cfbcf, However Inspection must be performed before merge can occur...

We must allow extra args. E.g., --ekat-kokkos-device N may be added by EKAT's test-launcher

… action Prevents errors when same workflow runs TAS action with different input machines

Both pull_request and pull_request_review must be in the same concurrency group

bartgol · 2024-10-31T23:24:05Z

I changed my mind regarding pull_request_review trigger. Here's my reasoning:

the trigger makes the checks different from those with pull_request trigger, effectively duplicating our checks for no reason
when a reviewer approves, it takes not much time to go and click on "re-run jobs" in the checks tab...

bartgol · 2024-11-01T01:18:49Z

Note: the gcc-cuda/opt build fails for a good reason: the wind speed diag unit test is basically comparing std::sqrt(u*u+v*v) as computed on host with sqrt(u*u+v*v) from the CUDA kernel. But CUDA does not guarantee bfb accuracy of sqrt between host and device, so we can't expect this to pass in general.

I will address this in another issue, and maybe do a separate PR.

Edit: actually, CUDA guide states that the max ulp error for sqrt (double) is 0. But that's assuming that the reference impl is obtained "according to the round-to-nearest ties-to-even rounding mode". According to my AI bot, GCC does comply with that mode, even when built with -O3.

I am debated on whether to merge this or fix the diff first... One mitigation possibility would be to use an approximate equality check...

Edit: I ended up switching to an approximate match check.

bartgol · 2024-11-01T03:25:19Z

@jgfouca I am going to merge without giving you a chance to re-review. Please, feel free to inspect the last two commits and request changes. They are however not relevant regarding the main part of the PR, which is enabling the CUDA build for AT2 for test-all-scream, and also removing the pull_request_review trigger, which backfired (it caused duplication of our testing for nothing).

Hence, I am merging to make sure other PRs can be tested for AT2 as well. I will address comments/cleanup in follow up PRs.

bartgol added the CI: workflow change approved Allow testing of PRs that alter a worfklow file label Oct 30, 2024

bartgol force-pushed the bartgol/eamxx/ghci-snl-cuda branch from d9209a4 to 6d3abef Compare October 30, 2024 23:53

bartgol added CI: workflow change approved Allow testing of PRs that alter a worfklow file and removed CI: workflow change approved Allow testing of PRs that alter a worfklow file labels Oct 30, 2024

bartgol added CI: workflow change approved Allow testing of PRs that alter a worfklow file and removed CI: workflow change approved Allow testing of PRs that alter a worfklow file labels Oct 31, 2024

bartgol added the AT: WIP label Oct 31, 2024

bartgol force-pushed the bartgol/eamxx/ghci-snl-cuda branch from 7aff00f to 1a4b331 Compare October 31, 2024 17:06

bartgol added CI: workflow change approved Allow testing of PRs that alter a worfklow file and removed CI: workflow change approved Allow testing of PRs that alter a worfklow file labels Oct 31, 2024

bartgol force-pushed the bartgol/eamxx/ghci-snl-cuda branch from 1a4b331 to ecdf7ce Compare October 31, 2024 19:48

bartgol added CI: workflow change approved Allow testing of PRs that alter a worfklow file and removed CI: workflow change approved Allow testing of PRs that alter a worfklow file labels Oct 31, 2024

bartgol self-assigned this Oct 31, 2024

bartgol requested review from jgfouca and tcclevenger October 31, 2024 20:12

tcclevenger previously approved these changes Oct 31, 2024

View reviewed changes

bartgol added 4 commits October 31, 2024 15:09

EAMxx: add support for ghci-snl-cuda in standalone testing

6dc8af9

Workflows: add support for ghci-snl-cuda also in eamxx-sa workflow

cc4a3e5

Workflows: skip eamxx v1/sa testing on draft PRs

cdde3e0

EAMxx: fix parsing of inputs in rrtmgp generate_baseline

0b5adb6

We must allow extra args. E.g., --ekat-kokkos-device N may be added by EKAT's test-launcher

bartgol force-pushed the bartgol/eamxx/ghci-snl-cuda branch from ecdf7ce to 0b5adb6 Compare October 31, 2024 21:09

Workflows: use different artifact names for different machines in TAS…

6df9583

… action Prevents errors when same workflow runs TAS action with different input machines

bartgol dismissed tcclevenger’s stale review via 6df9583 October 31, 2024 21:19

Workflows: fix concurrency group in eamxx workflows

7382c0c

Both pull_request and pull_request_review must be in the same concurrency group

jgfouca previously approved these changes Oct 31, 2024

View reviewed changes

bartgol added 2 commits October 31, 2024 17:21

Workflows: simplified concurrency of eamxx workflows

fc82112

Workflows: remove pull_request_review event from eamxx workflows

823b774

bartgol dismissed jgfouca’s stale review via 823b774 October 31, 2024 23:22

bartgol added CI: workflow change approved Allow testing of PRs that alter a worfklow file and removed CI: workflow change approved Allow testing of PRs that alter a worfklow file labels Oct 31, 2024

EAMxx: disable cuda-aware MPI in ghci-snl-cuda

3724d45

EAMxx: allow 1ulp diff in wind_speed test in release mode

5da88db

bartgol merged commit b1efc60 into master Nov 1, 2024
14 checks passed

bartgol deleted the bartgol/eamxx/ghci-snl-cuda branch November 1, 2024 03:25

EAMxx: add support for ghci-snl-cuda in standalone testing #3077

EAMxx: add support for ghci-snl-cuda in standalone testing #3077

Uh oh!

Conversation

bartgol commented Oct 30, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Uh oh!

E3SM-Bot commented Oct 31, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Uh oh!

E3SM-Bot commented Oct 31, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

E3SM-Bot commented Oct 31, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Uh oh!

E3SM-Bot commented Oct 31, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Uh oh!

E3SM-Bot commented Oct 31, 2024

Uh oh!

bartgol commented Oct 31, 2024

Uh oh!

bartgol commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bartgol commented Nov 1, 2024

Uh oh!

Uh oh!

Uh oh!

bartgol commented Nov 1, 2024 •

edited

Loading