Automatically Enumerate Dryruns and MpiOnlyExperiment

[michaelmckinsey1]

Description

• Supersedes [WIP] Automatically Enumerate Dryruns using CI Matrix #701. The ability to run newly added experiment.py's in CI without changes to any CI files (.github), using benchpark list which will automatically execute dryrun testing for:
  • New benchmarks.
  • New experiments on existing benchmarks
  • New modifiers
  • New systems
• Introduces new management class for experiments to inherit from MpiOnlyExperiment if running the experiment without a programming model is not valid, e.g. babelstream. This will return a descriptive error during experiment initialization, instead of the user getting a confusing error during build/run:

$ benchpark experiment init --dest babelstream babelstream
Traceback (most recent call last):
  File "/Users/mckinsey1/Documents/repos/benchpark/lib/main.py", line 336, in <module>
    exit_code = main()
                ^^^^^^
  File "/Users/mckinsey1/Documents/repos/benchpark/lib/main.py", line 104, in main
    exit_code = action(args)
                ^^^^^^^^^^^^
  File "/Users/mckinsey1/Documents/repos/benchpark/lib/benchpark/cmd/experiment.py", line 70, in command
    actions[args.experiment_subcommand](args)
  File "/Users/mckinsey1/Documents/repos/benchpark/lib/benchpark/cmd/experiment.py", line 20, in experiment_init
    experiment = experiment_spec.experiment
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mckinsey1/Documents/repos/benchpark/lib/benchpark/spec.py", line 380, in experiment
    return self.experiment_class(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mckinsey1/Documents/repos/benchpark/lib/benchpark/experiment.py", line 276, in __init__
    raise NotImplementedError(
NotImplementedError: "babelstream" cannot run with MPI only without inheriting from MpiOnlyExperiment. Choose from ['openmp', 'rocm', 'cuda']

• depends on Update benchpark list to query programming model #903
• depends on Fix Broken Test from Prepend Path #921
• Open question on whether it is necessary to enumerate variants like compiler. These only result in branching, which outside of building we can only check for syntax.
• Improve logs for failing dryruns

Adding/modifying core functionality, CI, or documentation:

• Update .github
• Update dryrun docs
• Update github dryrun developer docs once Update Nightly Testing Suite #892 merged

[codecov-commenter]

Codecov Report

❌ Patch coverage is 68.75000% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.75%. Comparing base (fc3e74f) to head (2e424f7).

Files with missing lines	Patch %	Lines
lib/benchpark/experiment.py	33.33%	4 Missing and 4 partials ⚠️
lib/benchpark/mpi.py	84.61%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #904      +/-   ##
===========================================
+ Coverage    41.09%   41.75%   +0.66%     
===========================================
  Files           23       24       +1     
  Lines         1555     1583      +28     
  Branches       225      231       +6     
===========================================
+ Hits           639      661      +22     
- Misses         871      872       +1     
- Partials        45       50       +5     

Flag	Coverage Δ
unittests	41.75% <68.75%> (+0.66%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[michaelmckinsey1]

Ready if CI passing