[WIP] Automatically Enumerate Dryruns using CI Matrix

[michaelmckinsey1]

Description

• Downsides:

  • All of the individual jobs clutter up the CI at the bottom of the PR (see checks tab for better visualization of jobs).
  • More difficult to enumerate specific run, e.g. llnl-cluster cluster=ruby compiler=not-default would require its own job

• The ability to run newly added experiment.py's in CI without changes to any CI files (.github), using benchpark list experiments (which will pick up any new experiments added to benchpark).

• Removes 975 lines of CI from .github/workflows/run.yml that we no longer need to maintain.

• CI completes much faster, as checks run in parallel instead of sequentially.

• This PR has caught errors with experiments that were not caught in develop CI
  step still, and require an update to the python script. Otherwise, the user could add one-off CI the same way as before.

• This only tests 4 systems. Perhaps the others could each get a manual unit test?

• Depends on Starting provides, requires directives #717.

• Update benchpark list systems to say which ones provide ProgrammingModel (cuda, rocm, openmp) Update benchpark list to query programming model #903

• Update benchpark list experiments to say which ones require ProgrammingModel

• Generate experiment lists semi-dynamically (for now, only want Livermore systems and benchmarks)

• Can we run multiple jobs in the same allocation?

• Related to Extend docs to specify checklist of what's needed for experiment #424

• Update docs

• Update .github/workflows and .gitlab/ci unit tests (if needed)

[michaelmckinsey1]

{
    "openmp": {
        "benchmark_spec": [
            "amg2023+openmp",
            "babelstream+openmp",
            "genesis+openmp",
            "gromacs+openmp",
            "hpcg+openmp",
            "hpl+openmp",
            "kripke+openmp",
            "lammps+openmp",
            "quicksilver+openmp",
            "qws+openmp",
            "raja-perf+openmp",
            "saxpy+openmp"
        ],
        "system_spec": [
            "llnl-cluster cluster=ruby"
        ]
    },
    "cuda": {
        "benchmark_spec": [
            "amg2023+cuda",
            "babelstream+cuda",
            "gromacs+cuda",
            "kripke+cuda",
            "laghos+cuda",
            "lammps+cuda",
            "osu-micro-benchmarks+cuda",
            "raja-perf+cuda",
            "remhos+cuda",
            "saxpy+cuda"
        ],
        "system_spec": [
            "llnl-sierra"
        ]
    },
    "rocm": {
        "benchmark_spec": [
            "amg2023+rocm",
            "babelstream+rocm",
            "gromacs+rocm",
            "kripke+rocm",
            "laghos+rocm",
            "lammps+rocm",
            "osu-micro-benchmarks+rocm",
            "raja-perf+rocm",
            "remhos+rocm",
            "saxpy+rocm"
        ],
        "system_spec": [
            "llnl-elcapitan cluster=tioga"
        ]
    },
    "weak": {
        "benchmark_spec": [
            "amg2023+weak",
            "hpcg+weak",
            "hpl+weak",
            "ior+weak",
            "kripke+weak",
            "quicksilver+weak"
        ],
        "system_spec": [
            "generic-x86"
        ]
    },
    "strong": {
        "benchmark_spec": [
            "amg2023+strong",
            "gpcnet+strong",
            "hpcg+strong",
            "hpl+strong",
            "ior+strong",
            "kripke+strong",
            "laghos+strong",
            "md-test+strong",
            "phloem+strong",
            "quicksilver+strong",
            "raja-perf+strong",
            "remhos+strong",
            "smb+strong"
        ],
        "system_spec": [
            "generic-x86"
        ]
    },
    "single_node": {
        "benchmark_spec": [
            "ad+single_node",
            "amg2023+single_node",
            "babelstream+single_node",
            "genesis+single_node",
            "gpcnet+single_node",
            "gromacs+single_node",
            "hpcg+single_node",
            "hpl+single_node",
            "ior+single_node",
            "kripke+single_node",
            "laghos+single_node",
            "lammps+single_node",
            "md-test+single_node",
            "osu-micro-benchmarks+single_node",
            "phloem+single_node",
            "quicksilver+single_node",
            "qws+single_node",
            "raja-perf+single_node",
            "remhos+single_node",
            "saxpy+single_node",
            "smb+single_node",
            "stream+single_node"
        ],
        "system_spec": [
            "generic-x86"
        ]
    },
    "throughput": {
        "benchmark_spec": [
            "amg2023+throughput",
            "kripke+throughput"
        ],
        "system_spec": [
            "generic-x86"
        ]
    }
}

Example JSON file created by the new script. For each variant, a separate job is started for each benchmark_spec and system_spec combination.

[michaelmckinsey1]

We will probably want to decouple the changes in .github from the experiments in a separate PR. I am using this PR to illustrate as develop CI does not catch this issues.

Being addressed in #714

[michaelmckinsey1]

• generate systems dynamically instead of hardcoding systems. Need benchpark list systems -e cuda or something
• How many jobs can we run at the same time i.e. If we matrix every experiment with every system.
  • 256 jobs max
• How to dynamically handle different system compiler variants? Does this matter for dryruns?

[michaelmckinsey1]

Depends on #714 to fix unit tests

[pearce8]

@michaelmckinsey1 please take a look at #717 and let me know how you could use it in this PR.

[michaelmckinsey1]

@michaelmckinsey1 please take a look at #717 and let me know how you could use it in this PR.

@pearce8 with parent classes we could update the benchpark list systems command to work similarly to benchpark list experiments where we can query for programming model. We could then query for a cuda system, etc. without hardcoding the mapping in .github/workflows/gen-dynamic-runs.py.

[pearce8]

@michaelmckinsey1 Please resolve conflicts, and lets talk about what needs to be done in this PR vs. later.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 41.40%. Comparing base (eb33dca) to head (f7f09f6).
Report is 42 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #701      +/-   ##
===========================================
+ Coverage    40.30%   41.40%   +1.09%     
===========================================
  Files           15       22       +7     
  Lines         1119     1512     +393     
  Branches       152      214      +62     
===========================================
+ Hits           451      626     +175     
- Misses         632      841     +209     
- Partials        36       45       +9     

Flag	Coverage Δ
unittests	41.40% <ø> (+1.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[michaelmckinsey1]

Superseded by #904