Skip to content

Conversation

jsign
Copy link
Collaborator

@jsign jsign commented Sep 24, 2025

This PR's primary goal is to leverage the work done in the integration test pipeline to generate an automated website displaying zkVM execution results.

Summary of changes:

  • The integration tests allow a new optional WORKLOAD_OUTPUT_DIR so the test results aren't discarded, but saved to a desired path. The way it works is that it uses tempdir (as it always did yesterday) or a defined path depending on that var.
  • In CI, we use this new WORKLOAD_OUTPUT_DIR so all the integration test results runs are uploaded as artifacts. This can be useful to further inspect runs, or have a medium-term historical view if useful.
  • After integration tests are run, a new GH job collects all uploaded artifacts, it merges them (since integration tests are run in parallel for different EL/zkVM combinations), and generates a website using a Python script. Then, another job after deploys this website.
  • For the stateless-validator guest program runs, the zkevm-metrics folder structure has a new level "EL". So instead of now being zkevm-metrics/<zkvm>/[results] is zkevm-metrics/<el>/<zkvm>/[results]. Apart for being more organized, it was useful to allow the generated website to identify ELs correctly.
  • I reorganized the integration test jobs extracting runs in a separate component, which avoids some duplication that was present and potential future reuse.

The idea is to try to generate execution reports automatically:

  • on every merge to master, since we can get that somewhat for free, since we were already running integration tests. Today this covers 30 mainnet blocks, invalid block, and empty block, but anything else we add in the future will get included easily.
  • I'll experiment with whether we have a separate daily runner for more EEST cases. I'm not sure the CI machines can run all of them. If it is too heavy, I could manually choose the already identified worst cases to at least keep track of those most important ones and see which zkVM crash or are too slow (which is actually the most important signal we want to track).

https://eth-act.github.io/zkevm-benchmark-workload/

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
…eless-validator guest program

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
Copy link
Collaborator Author

@jsign jsign Sep 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the integration test refactoring, as a side effect, the CI runs are a bit better organized in the UI:
image

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pipeline:
image

Comment on lines +61 to +62
- zkvm: openvm
threads: 2
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an old setup since OpenVM uses quite a lot of RAM per case, so going very far in parallelization makes the CI machine struggle. But since OpenVM emulation is very fast, doesn't hurt much CI duration.

For the rest, I configured 12 threads to avoid going with 16 which is the default for the CI machine. Mainly to give some stability to wall-clock time runs.

run: |
${{ matrix.zkvm == 'openvm' && 'RAYON_NUM_THREADS=1' || '' }} RUST_LOG=warn,benchmark_runner=info ZKVM=${{ matrix.zkvm }} EL=${{ matrix.el }} cargo test --release -p integration-tests -- --test-threads=1 ${{ matrix.test }}
zkvm: zisk # See https://github.com/eth-act/ere/issues/186
uses: ./.github/workflows/run-benchmark.yml
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of deleted code in this file was extracted to this separate component for better modularity and avoid repetition.

el: 'none'
threads: 12

generate-benchmark-website:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and next jobs are new ones pulling uploaded artifacts, merging, generating website and publishing.

Pending to only run this on master -- for now is always running to test the PR.

)?;
let config = RunConfig {
output_folder: cli.output_folder,
sub_folder: Some(el.as_ref().to_lowercase()),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to eventually reconsider this solution, but probably closer when we accept raw-ELFs potentially for externally compiled ELFs.

@jsign jsign changed the title Automatic execution website for integration test cases executions Automatic website gen for integration test cases executions Sep 24, 2025
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
@jsign jsign marked this pull request as ready for review September 24, 2025 15:58
@jsign jsign merged commit a83e6bd into master Sep 24, 2025
44 of 45 checks passed
@jsign jsign deleted the jsign-gen-page branch September 24, 2025 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant