Add SCREAMv1 test to e3sm_gpucxx suite #5495

brhillman · 2023-03-02T15:39:43Z

Add F2010-SCREAMv1 test to e3sm_gpucxx suite to get test coverage on GPU for
EAMxx codebase in main E3SM repo.

[BFB]

xyuan

I have tested on Summit using both cpu and gpu, and it works fine, thanks.

amametjanov · 2023-03-04T00:48:41Z

./cime/scripts/create_test ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1 --compiler gnugpu

with latest master on Summit runs into bld errors:

components/eamxx/src/dynamics/homme/atmosphere_dynamics.cpp(817): error: class "Homme::SimulationParams" has no member "rearth"
/autofs/nccs-svm1_sw/summit/gcc/9.1.0-alpha+20190716/include/c++/9.1.0/ext/new_allocator.h(147): error: no instance of constructor "Homme::Diagnostics::Diagnostics" matches the argument list
            argument types are: (int, __nv_bool)

On Crusher with --compiler crayclanggpu:

externals/ekat/extern/kokkos/core/src/../../tpls/desul/include/desul/atomics/Generic.hpp:618:9: error: call to 'atomic_fetch_add' is ambiguous
  (void)atomic_fetch_add(dest, val, order, scope);
        ^~~~~~~~~~~~~~~~

Does this look familiar in runs of this case with SCREAM repo?

brhillman · 2023-03-07T14:50:03Z

Thanks @amametjanov this actually passes for me on the SCREAM repo, and confirmed it fails with a local master merge into E3SM on summit. I'll try to figure out what's going on with the master merge.

brhillman · 2023-03-07T15:20:30Z

Okay, I see what happened. It looks like #5481 changed some things in hommexx that haven't been fixed upstream in E3SM yet. The fixes are on the SCREAM repo, so I will need to open another PR to pull those into E3SM I think, or pull those into this PR (looks like the diffs only affect sources in the eamxx directory, plus shoc).

rljacob

There should not be 356 commits for such a simple addition.

brhillman · 2023-04-05T22:02:04Z

@rljacob I had to pull in a scream->e3sm merge because there were conflicts between the two repos since this PR was opened that prevented SCREAMv1 from building on the E3SM repo. I can move this to a separate PR, or rename this one to reflect this. Was waiting on my tests to update this PR (which just passed).

rljacob · 2023-04-05T22:59:16Z

Please make a separate PR.

rljacob · 2023-04-05T23:22:57Z

If you need to resync SCREAM with E3SM and its 300+ commits, that should be its own PR.

brhillman · 2023-04-05T23:28:22Z

This requires #5582 to fix SCREAMv1 build errors.

brhillman · 2023-04-05T23:30:43Z

If you need to resync SCREAM with E3SM and its 300+ commits, that should be its own PR.

Done. My reasoning here was to consider this PR the upstream merge, and as a side effect add the test to make sure it doesn’t break again, rather than considering the upstream merge the effect. In any event, #5582 is opened to bring EAMxx up to date, then this PR will just add the test.

(PR #5495) Add SCREAMv1 test to e3sm_gpucxx suite Add F2010-SCREAMv1 test to e3sm_gpucxx suite to get test coverage on GPU for EAMxx codebase in main E3SM repo. [BFB]

amametjanov · 2023-05-26T02:14:35Z

Do we need to add machine-specific cmake in components/eamxx/cmake/machine-files/?
From a JLSE run:

-- Found scream component
No macro file found: /gpfs/jlse-fs0/projects/climate/azamat/scratch/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.jlse_oneapi-ifxgpu.JNextGpucxx20230525_060459/cmake_macros/LINUX.cmake
No macro file found: /gpfs/jlse-fs0/projects/climate/azamat/scratch/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.jlse_oneapi-ifxgpu.JNextGpucxx20230525_060459/cmake_macros/jlse.cmake
No macro file found: /gpfs/jlse-fs0/projects/climate/azamat/scratch/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.jlse_oneapi-ifxgpu.JNextGpucxx20230525_060459/cmake_macros/oneapi-ifxgpu_LINUX.cmake
No macro file found: /gpfs/jlse-fs0/projects/climate/azamat/scratch/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.jlse_oneapi-ifxgpu.JNextGpucxx20230525_060459/cmake_macros/oneapi-ifxgpu_jlse.cmake
CMake Error at cmake/build_eamxx.cmake:39 (include):
  include could not find requested file:

    /gpfs/jlse-fs0/projects/climate/testing/E3SM/components/eamxx/cmake/machine-files/jlse.cmake
Call Stack (most recent call first):
  CMakeLists.txt:120 (build_eamxx)

Similar error on Crusher: https://my.cdash.org/viewTest.php?buildid=2341352

Works on Ascent: https://my.cdash.org/viewTest.php?buildid=2341401

rljacob · 2023-07-06T17:34:04Z

notes: needs some syncing and restesting. @brhillman will do.

rljacob · 2023-07-12T05:15:03Z

@brhillman is this ready now?

amametjanov · 2023-07-14T22:28:13Z

If I copy crusher-scream-gpu.cmake to components/eamxx/cmake/machine-files/crusher.cmake, the build succeeds, but the run fails during atm-init with

$ tail /lustre/orion/cli115/proj-shared/azamat/e3sm_scratch/crusher/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.crusher_crayclanggpu.G.20230714-chk-new-scream-gputest4/run/e3sm.log.350998.230714-173930 
0: (seq_comm_printcomms)    31     0     1     1  ALLESPID:
0: (seq_comm_printcomms)    32     0     8     1  CPLALLESPID:
0: (seq_comm_printcomms)    33     0     1     1  ESP:
0: (seq_comm_printcomms)    34     0     8     1  CPLESP:
0: (seq_comm_printcomms)    35     0     1     1  ALLIACID:
0: (seq_comm_printcomms)    36     0     8     1  CPLALLIACID:
0: (seq_comm_printcomms)    37     0     1     1  IAC:
0: (seq_comm_printcomms)    38     0     8     1  CPLIAC:
srun: error: crusher188: tasks 0-7: Bus error
srun: Terminating StepId=350998.0

sarats · 2023-07-16T16:26:37Z

Just FYI.
Testing with PR #5745 that removes the CNL.cmake file, https://my.cdash.org/test/85523812 results in errors:

Essentially, looking for non-existent macro files (Linux.cmake, crusher.cmake and crayclanggpu_Linux.cmake).

165 No macro file found: /lustre/orion/cli115/proj-shared/sarat/e3sm_scratch/crusher/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.crusher_crayclanggpu.C.JNextGpucxx20230714_154451/cmake_macros/Linux.cmake
166 No macro file found: /lustre/orion/cli115/proj-shared/sarat/e3sm_scratch/crusher/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.crusher_crayclanggpu.C.JNextGpucxx20230714_154451/cmake_macros/crusher.cmake
167 No macro file found: /lustre/orion/cli115/proj-shared/sarat/e3sm_scratch/crusher/J/ERP_Ln9.ne4pg2_ne4pg2.F2010-SCREAMv1.crusher_crayclanggpu.C.JNextGpucxx20230714_154451/cmake_macros/crayclanggpu_Linux.cma    ke

jgfouca · 2024-05-15T22:02:35Z

@rljacob , there are a few ways. This new test in the suite e3sm_gpucxx which does not have "time" field. We could add this field and set it to something long enough to give it a chance to finish. As an alternative, if this test is only too slow on one platform, we can go to that machine, run the test by hand ./create_test $test --walltime=4:00:00. If it passes, it will store the runtime under the $baseline/walltimes area and our "smart" walltime system will give that time the highest precedence when choosing a default time.

rljacob · 2024-05-15T22:27:43Z

Lets do the second option but wait until pm comes back online.

ndkeen · 2024-05-15T23:15:22Z

Note that when I try this same test on muller-gpu (generally same as pm-gpu), I get an error and a hang. We would not expect any ne4 test to take much time, so maybe don't increase the walltime for the test.

2: (GTL DEBUG: 2) cuIpcGetMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE, line no 97

Checking out scream instead (again on muller-gpu) and these tests work fine. So must be something we are doing differently.

jgfouca · 2024-05-17T19:47:47Z

Should I merge to master or hold off?

rljacob · 2024-05-29T02:29:03Z

@ndkeen is the one SCREAM test in this suite still running out time?

ndkeen · 2024-05-29T03:00:11Z

If I recall, I think we found some issue with the test hanging.

When I tried it just now, I get a build error:

components/eamxx/src/share/atm_process/atmosphere_process_dag.cpp(149): error: class "ekat::units::Units" has no member "get_string"
          s += " [" + fid.get_units().get_string() + "]";
                                      ^

rljacob · 2024-05-29T17:48:11Z

@bartgol since Jim is on vacation, could you take a look at the above build error on pm-gpu?

bartgol · 2024-05-30T15:08:26Z

It seems like the ekat submodule is not recent enough. However, it is recent enough in e3sm/master. Perhaps the branch is old, and is forcing a roll back on the ekat submodule? I'll try merging into next locally now, to see what happens to the submodule.

bartgol · 2024-05-30T15:12:21Z

Oh, wait, this has the new ekat, but an old EAMxx. EAMxx is still using code from an old ekat. We need to update eamxx first. Perhaps the cleanest thing is to remove the branch from next, hard reset the branch to track an up-to-date eamxx, and then remerge to next.

rljacob · 2024-05-30T15:47:47Z

I'd rather have an eamxx update in a separate PR. Then when that is on next, this should work.

bartgol · 2024-05-30T16:50:41Z

That's fine with me. And now, let's draw straws! :)

Jokes aside, how soon do we want this to be integrated?

rljacob · 2024-06-03T21:43:09Z

I'm really sick of seeing this open PR so an eamxx downstream merge soon would be good. But I think there is an upstream still pending over in scream. @jgfouca can you push that along?

bartgol · 2024-06-10T20:49:44Z

@rljacob the downstream merge in the scream repo should move along today. There was a failing test due to a bad auto-resolution of a conflict. Should be fixed now, and tests should be passing on the next AT round (mappy is a bit backed up due to it being offline for a few days).

rljacob · 2024-07-13T00:43:45Z

After the emaxx downstream merge in #6489 there's no longer a kokkos build error but there is an error from COSP. @jgfouca can you look at it next week?

mahf708 · 2024-07-25T17:43:16Z

@rljacob, this scream build of cosp was broken due to: #6407

rljacob · 2024-08-07T03:42:37Z

@zyuying can you look at the build error from your COSP changes when building scream on pm-gpu?

github-actions · 2024-08-21T16:44:23Z

PR Preview Action v1.4.7
🚀 Deployed preview to https://E3SM-Project.github.io/E3SM/pr-preview/pr-5495/
on branch `gh-pages` at 2024-08-21 16:44 UTC

brhillman · 2024-08-21T17:16:58Z

@rljacob this should now fix the SCREAMv1 build error associated with the COSP updates.

… (PR #5495) Add F2010-SCREAMv1 test to e3sm_gpucxx suite to get test coverage on GPU for EAMxx codebase in main E3SM repo. Fixes #6540. [BFB]

rljacob · 2024-08-22T16:16:42Z

It passed! on pm-gpu

brhillman added BFB PR leaves answers BFB Testing Anything related to unit/system tests EAMxx Issues related to EAMxx labels Mar 2, 2023

brhillman requested review from xyuan and rljacob March 2, 2023 15:39

rljacob assigned amametjanov Mar 2, 2023

xyuan approved these changes Mar 3, 2023

View reviewed changes

rljacob approved these changes Mar 14, 2023

View reviewed changes

brhillman force-pushed the brhillman/cime_config/add-scream-gpucxx-test branch from 979b436 to e2caf84 Compare April 3, 2023 17:45

rljacob requested changes Apr 5, 2023

View reviewed changes

brhillman changed the title ~~Add F2010-SCREAMv1 test to e3sm_gpucxx suite~~ Fix F2010-SCREAMv1 build and add test to e3sm_gpucxx suite Apr 5, 2023

brhillman force-pushed the brhillman/cime_config/add-scream-gpucxx-test branch from 9d34608 to 5111a4f Compare April 5, 2023 23:26

brhillman changed the title ~~Fix F2010-SCREAMv1 build and add test to e3sm_gpucxx suite~~ Add test to e3sm_gpucxx suite Apr 5, 2023

brhillman changed the title ~~Add test to e3sm_gpucxx suite~~ Add SCREAMv1 test to e3sm_gpucxx suite Apr 5, 2023

rljacob approved these changes Apr 5, 2023

View reviewed changes

brhillman force-pushed the brhillman/cime_config/add-scream-gpucxx-test branch from 5111a4f to 13d1962 Compare July 6, 2023 22:27

mahf708 mentioned this pull request Jul 26, 2024

set track_avg_cnt: true by default E3SM-Project/scream#2922

Closed

mahf708 mentioned this pull request Aug 8, 2024

Build error with basic ne30 scream case on pm-gpu SMS_Ld1.ne30_ne30.F2010-SCREAMv1.pm-gpu_gnugpu #6540

Closed

ndkeen added the COSP label Aug 8, 2024

Add F2010-SCREAMv1 test to e3sm_gpucxx suite

ec98339

brhillman force-pushed the brhillman/cime_config/add-scream-gpucxx-test branch from 13d1962 to ec98339 Compare August 21, 2024 16:39

Revert to external cosp.F90 to fix build error for EAMxx

b326f27

rljacob assigned rljacob and unassigned jgfouca Aug 21, 2024

rljacob merged commit 37583bd into master Aug 22, 2024
21 checks passed

rljacob deleted the brhillman/cime_config/add-scream-gpucxx-test branch August 22, 2024 16:18

Add SCREAMv1 test to e3sm_gpucxx suite #5495

Add SCREAMv1 test to e3sm_gpucxx suite #5495

Uh oh!

Conversation

brhillman commented Mar 2, 2023 • edited by rljacob Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xyuan left a comment

Choose a reason for hiding this comment

Uh oh!

amametjanov commented Mar 4, 2023

Uh oh!

brhillman commented Mar 7, 2023

Uh oh!

brhillman commented Mar 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rljacob left a comment

Choose a reason for hiding this comment

Uh oh!

brhillman commented Apr 5, 2023

Uh oh!

rljacob commented Apr 5, 2023

Uh oh!

rljacob commented Apr 5, 2023

Uh oh!

brhillman commented Apr 5, 2023

Uh oh!

brhillman commented Apr 5, 2023

Uh oh!

amametjanov commented May 26, 2023

Uh oh!

rljacob commented Jul 6, 2023

Uh oh!

rljacob commented Jul 12, 2023

Uh oh!

amametjanov commented Jul 14, 2023

Uh oh!

sarats commented Jul 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgfouca commented May 15, 2024

Uh oh!

rljacob commented May 15, 2024

Uh oh!

ndkeen commented May 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgfouca commented May 17, 2024

Uh oh!

rljacob commented May 29, 2024

Uh oh!

ndkeen commented May 29, 2024

Uh oh!

rljacob commented May 29, 2024

Uh oh!

bartgol commented May 30, 2024

Uh oh!

bartgol commented May 30, 2024

Uh oh!

rljacob commented May 30, 2024

Uh oh!

bartgol commented May 30, 2024

Uh oh!

rljacob commented Jun 3, 2024

Uh oh!

bartgol commented Jun 10, 2024

Uh oh!

rljacob commented Jul 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahf708 commented Jul 25, 2024

Uh oh!

rljacob commented Aug 7, 2024

Uh oh!

github-actions bot commented Aug 21, 2024

Uh oh!

brhillman commented Aug 21, 2024

Uh oh!

rljacob commented Aug 22, 2024

brhillman commented Mar 2, 2023 •

edited by rljacob

Loading

brhillman commented Mar 7, 2023 •

edited

Loading

sarats commented Jul 16, 2023 •

edited

Loading

ndkeen commented May 15, 2024 •

edited

Loading

rljacob commented Jul 13, 2024 •

edited

Loading