Measure code-size #1089

willieyz · 2025-06-18T06:51:05Z

Resolves Measure code-size #888
Resolves Add script to measure code size #889

This PR mean to add the code size measurement for each cycle benchmark for MLKEM binaries.

We adds the ability to measure code size for tests and examples:

For tests: run ./scripts/tests size
For examples: run ./scripts/tests examples -l <example_name> --verbose

The function is implemented using the size command:

Adds the size command and corresponding Makefile configuration for both tests and each example.
Filter out empty size results to keep the output clean.
Sort results in descending order by size to enhance readability.

This enhancement helps track the binary size alongside performance metrics to better understand trade-offs.

Makefile

oqs-bot

Mac Mini (M1, 2020) benchmarks

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`12302` cycles	`12302` cycles	`1`
`ML-KEM-512 encaps`	`14893` cycles	`14894` cycles	`1.00`
`ML-KEM-512 decaps`	`19241` cycles	`19240` cycles	`1.00`
`ML-KEM-768 keypair`	`21358` cycles	`21360` cycles	`1.00`
`ML-KEM-768 encaps`	`23566` cycles	`23566` cycles	`1`
`ML-KEM-768 decaps`	`30140` cycles	`30140` cycles	`1`
`ML-KEM-1024 keypair`	`30338` cycles	`30337` cycles	`1.00`
`ML-KEM-1024 encaps`	`34664` cycles	`34663` cycles	`1.00`
`ML-KEM-1024 decaps`	`44158` cycles	`44159` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A76 (Raspberry Pi 5) benchmarks

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`28819` cycles	`28846` cycles	`1.00`
`ML-KEM-512 encaps`	`34030` cycles	`34092` cycles	`1.00`
`ML-KEM-512 decaps`	`44614` cycles	`44555` cycles	`1.00`
`ML-KEM-768 keypair`	`49253` cycles	`49191` cycles	`1.00`
`ML-KEM-768 encaps`	`54275` cycles	`54241` cycles	`1.00`
`ML-KEM-768 decaps`	`69122` cycles	`69155` cycles	`1.00`
`ML-KEM-1024 keypair`	`71572` cycles	`71609` cycles	`1.00`
`ML-KEM-1024 encaps`	`79935` cycles	`79844` cycles	`1.00`
`ML-KEM-1024 decaps`	`99938` cycles	`99966` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`9551` cycles	`9530` cycles	`1.00`
`ML-KEM-512 encaps`	`11044` cycles	`11002` cycles	`1.00`
`ML-KEM-512 decaps`	`15157` cycles	`15201` cycles	`1.00`
`ML-KEM-768 keypair`	`16338` cycles	`16335` cycles	`1.00`
`ML-KEM-768 encaps`	`17734` cycles	`17744` cycles	`1.00`
`ML-KEM-768 decaps`	`23449` cycles	`24249` cycles	`0.97`
`ML-KEM-1024 keypair`	`21901` cycles	`21869` cycles	`1.00`
`ML-KEM-1024 encaps`	`24048` cycles	`24003` cycles	`1.00`
`ML-KEM-1024 decaps`	`31730` cycles	`31594` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 4th gen (c7i) (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`28682` cycles	`28746` cycles	`1.00`
`ML-KEM-512 encaps`	`34635` cycles	`34636` cycles	`1.00`
`ML-KEM-512 decaps`	`44383` cycles	`44512` cycles	`1.00`
`ML-KEM-768 keypair`	`48071` cycles	`48144` cycles	`1.00`
`ML-KEM-768 encaps`	`55872` cycles	`55930` cycles	`1.00`
`ML-KEM-768 decaps`	`67360` cycles	`67518` cycles	`1.00`
`ML-KEM-1024 keypair`	`73244` cycles	`73211` cycles	`1.00`
`ML-KEM-1024 encaps`	`84768` cycles	`85038` cycles	`1.00`
`ML-KEM-1024 decaps`	`99522` cycles	`99295` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A55 (Snapdragon 888) benchmarks

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`59480` cycles	`59549` cycles	`1.00`
`ML-KEM-512 encaps`	`66725` cycles	`66677` cycles	`1.00`
`ML-KEM-512 decaps`	`85390` cycles	`85415` cycles	`1.00`
`ML-KEM-768 keypair`	`101484` cycles	`101533` cycles	`1.00`
`ML-KEM-768 encaps`	`112765` cycles	`112580` cycles	`1.00`
`ML-KEM-768 decaps`	`139355` cycles	`139544` cycles	`1.00`
`ML-KEM-1024 keypair`	`154110` cycles	`154359` cycles	`1.00`
`ML-KEM-1024 encaps`	`170940` cycles	`174595` cycles	`0.98`
`ML-KEM-1024 decaps`	`207211` cycles	`210406` cycles	`0.98`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`17117` cycles	`17182` cycles	`1.00`
`ML-KEM-512 encaps`	`18880` cycles	`18950` cycles	`1.00`
`ML-KEM-512 decaps`	`24348` cycles	`24378` cycles	`1.00`
`ML-KEM-768 keypair`	`29408` cycles	`29441` cycles	`1.00`
`ML-KEM-768 encaps`	`30713` cycles	`30658` cycles	`1.00`
`ML-KEM-768 decaps`	`38521` cycles	`38480` cycles	`1.00`
`ML-KEM-1024 keypair`	`42744` cycles	`42755` cycles	`1.00`
`ML-KEM-1024 encaps`	`45025` cycles	`45074` cycles	`1.00`
`ML-KEM-1024 decaps`	`55522` cycles	`55414` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`11535` cycles	`11528` cycles	`1.00`
`ML-KEM-512 encaps`	`13281` cycles	`13269` cycles	`1.00`
`ML-KEM-512 decaps`	`18060` cycles	`18064` cycles	`1.00`
`ML-KEM-768 keypair`	`20135` cycles	`20154` cycles	`1.00`
`ML-KEM-768 encaps`	`21118` cycles	`21131` cycles	`1.00`
`ML-KEM-768 decaps`	`28892` cycles	`29291` cycles	`0.99`
`ML-KEM-1024 keypair`	`26980` cycles	`26917` cycles	`1.00`
`ML-KEM-1024 encaps`	`29125` cycles	`29010` cycles	`1.00`
`ML-KEM-1024 decaps`	`38671` cycles	`38548` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`28860` cycles	`28835` cycles	`1.00`
`ML-KEM-512 encaps`	`34104` cycles	`34002` cycles	`1.00`
`ML-KEM-512 decaps`	`44592` cycles	`44614` cycles	`1.00`
`ML-KEM-768 keypair`	`49196` cycles	`49218` cycles	`1.00`
`ML-KEM-768 encaps`	`54245` cycles	`54289` cycles	`1.00`
`ML-KEM-768 decaps`	`69146` cycles	`69153` cycles	`1.00`
`ML-KEM-1024 keypair`	`71623` cycles	`71493` cycles	`1.00`
`ML-KEM-1024 encaps`	`79860` cycles	`79874` cycles	`1.00`
`ML-KEM-1024 decaps`	`99990` cycles	`100023` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`16160` cycles	`16157` cycles	`1.00`
`ML-KEM-512 encaps`	`18261` cycles	`18261` cycles	`1`
`ML-KEM-512 decaps`	`24761` cycles	`24774` cycles	`1.00`
`ML-KEM-768 keypair`	`27726` cycles	`27712` cycles	`1.00`
`ML-KEM-768 encaps`	`30913` cycles	`29454` cycles	`1.05`
`ML-KEM-768 decaps`	`39896` cycles	`38985` cycles	`1.02`
`ML-KEM-1024 keypair`	`37557` cycles	`37565` cycles	`1.00`
`ML-KEM-1024 encaps`	`40525` cycles	`40532` cycles	`1.00`
`ML-KEM-1024 decaps`	`53054` cycles	`53090` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Intel Xeon 3rd gen (c6i)'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.03.

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-768 encaps`	`30913` cycles	`29454` cycles	`1.05`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`17937` cycles	`17904` cycles	`1.00`
`ML-KEM-512 encaps`	`21029` cycles	`20986` cycles	`1.00`
`ML-KEM-512 decaps`	`27642` cycles	`27630` cycles	`1.00`
`ML-KEM-768 keypair`	`30882` cycles	`30909` cycles	`1.00`
`ML-KEM-768 encaps`	`33545` cycles	`33574` cycles	`1.00`
`ML-KEM-768 decaps`	`43119` cycles	`43155` cycles	`1.00`
`ML-KEM-1024 keypair`	`44636` cycles	`44636` cycles	`1`
`ML-KEM-1024 encaps`	`49619` cycles	`49589` cycles	`1.00`
`ML-KEM-1024 decaps`	`62625` cycles	`62608` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 3rd gen (c6a) (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`38486` cycles	`38285` cycles	`1.01`
`ML-KEM-512 encaps`	`46797` cycles	`46822` cycles	`1.00`
`ML-KEM-512 decaps`	`60238` cycles	`60252` cycles	`1.00`
`ML-KEM-768 keypair`	`63633` cycles	`63625` cycles	`1.00`
`ML-KEM-768 encaps`	`74227` cycles	`74270` cycles	`1.00`
`ML-KEM-768 decaps`	`92227` cycles	`92226` cycles	`1.00`
`ML-KEM-1024 keypair`	`94073` cycles	`94098` cycles	`1.00`
`ML-KEM-1024 encaps`	`108047` cycles	`107847` cycles	`1.00`
`ML-KEM-1024 decaps`	`130692` cycles	`130713` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

AMD EPYC 4th gen (c7a) (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`36189` cycles	`36231` cycles	`1.00`
`ML-KEM-512 encaps`	`42808` cycles	`42825` cycles	`1.00`
`ML-KEM-512 decaps`	`55846` cycles	`55867` cycles	`1.00`
`ML-KEM-768 keypair`	`59906` cycles	`59928` cycles	`1.00`
`ML-KEM-768 encaps`	`68241` cycles	`68183` cycles	`1.00`
`ML-KEM-768 decaps`	`85808` cycles	`85653` cycles	`1.00`
`ML-KEM-1024 keypair`	`87532` cycles	`87662` cycles	`1.00`
`ML-KEM-1024 encaps`	`99674` cycles	`99620` cycles	`1.00`
`ML-KEM-1024 decaps`	`121324` cycles	`121262` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`19088` cycles	`19086` cycles	`1.00`
`ML-KEM-512 encaps`	`22301` cycles	`22298` cycles	`1.00`
`ML-KEM-512 decaps`	`29550` cycles	`29556` cycles	`1.00`
`ML-KEM-768 keypair`	`32590` cycles	`32597` cycles	`1.00`
`ML-KEM-768 encaps`	`35637` cycles	`35696` cycles	`1.00`
`ML-KEM-768 decaps`	`45966` cycles	`46013` cycles	`1.00`
`ML-KEM-1024 keypair`	`46883` cycles	`46884` cycles	`1.00`
`ML-KEM-1024 encaps`	`52191` cycles	`52153` cycles	`1.00`
`ML-KEM-1024 decaps`	`66153` cycles	`66037` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Intel Xeon 3rd gen (c6i) (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`46284` cycles	`46306` cycles	`1.00`
`ML-KEM-512 encaps`	`54619` cycles	`54597` cycles	`1.00`
`ML-KEM-512 decaps`	`70140` cycles	`70037` cycles	`1.00`
`ML-KEM-768 keypair`	`75302` cycles	`75413` cycles	`1.00`
`ML-KEM-768 encaps`	`86369` cycles	`86370` cycles	`1.00`
`ML-KEM-768 decaps`	`106567` cycles	`106475` cycles	`1.00`
`ML-KEM-1024 keypair`	`110891` cycles	`110983` cycles	`1.00`
`ML-KEM-1024 encaps`	`125150` cycles	`124907` cycles	`1.00`
`ML-KEM-1024 decaps`	`150478` cycles	`150576` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton4 (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`35794` cycles	`35802` cycles	`1.00`
`ML-KEM-512 encaps`	`40683` cycles	`40661` cycles	`1.00`
`ML-KEM-512 decaps`	`52067` cycles	`52073` cycles	`1.00`
`ML-KEM-768 keypair`	`59563` cycles	`59558` cycles	`1.00`
`ML-KEM-768 encaps`	`66482` cycles	`66620` cycles	`1.00`
`ML-KEM-768 decaps`	`81110` cycles	`81056` cycles	`1.00`
`ML-KEM-1024 keypair`	`88498` cycles	`88514` cycles	`1.00`
`ML-KEM-1024 encaps`	`98511` cycles	`98566` cycles	`1.00`
`ML-KEM-1024 decaps`	`117308` cycles	`117300` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton2 (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`59295` cycles	`59302` cycles	`1.00`
`ML-KEM-512 encaps`	`67994` cycles	`67892` cycles	`1.00`
`ML-KEM-512 decaps`	`86695` cycles	`86596` cycles	`1.00`
`ML-KEM-768 keypair`	`98779` cycles	`98784` cycles	`1.00`
`ML-KEM-768 encaps`	`110026` cycles	`109968` cycles	`1.00`
`ML-KEM-768 decaps`	`135194` cycles	`135068` cycles	`1.00`
`ML-KEM-1024 keypair`	`148624` cycles	`148295` cycles	`1.00`
`ML-KEM-1024 encaps`	`163830` cycles	`163695` cycles	`1.00`
`ML-KEM-1024 decaps`	`195527` cycles	`195150` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Graviton3 (no-opt)

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`38954` cycles	`38945` cycles	`1.00`
`ML-KEM-512 encaps`	`44867` cycles	`44916` cycles	`1.00`
`ML-KEM-512 decaps`	`56772` cycles	`56693` cycles	`1.00`
`ML-KEM-768 keypair`	`64328` cycles	`64399` cycles	`1.00`
`ML-KEM-768 encaps`	`71549` cycles	`71664` cycles	`1.00`
`ML-KEM-768 decaps`	`87733` cycles	`88068` cycles	`1.00`
`ML-KEM-1024 keypair`	`95556` cycles	`95502` cycles	`1.00`
`ML-KEM-1024 encaps`	`106211` cycles	`106330` cycles	`1.00`
`ML-KEM-1024 decaps`	`126893` cycles	`126833` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

Arm Cortex-A72 (Raspberry Pi 4) benchmarks

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`52051` cycles	`52688` cycles	`0.99`
`ML-KEM-512 encaps`	`60306` cycles	`60565` cycles	`1.00`
`ML-KEM-512 decaps`	`77580` cycles	`77447` cycles	`1.00`
`ML-KEM-768 keypair`	`89006` cycles	`88674` cycles	`1.00`
`ML-KEM-768 encaps`	`97033` cycles	`97261` cycles	`1.00`
`ML-KEM-768 decaps`	`120815` cycles	`121443` cycles	`0.99`
`ML-KEM-1024 keypair`	`133343` cycles	`133270` cycles	`1.00`
`ML-KEM-1024 encaps`	`146295` cycles	`145640` cycles	`1.00`
`ML-KEM-1024 decaps`	`178640` cycles	`179743` cycles	`0.99`

This comment was automatically generated by workflow using github-action-benchmark.

oqs-bot

SpacemiT K1 8 (Banana Pi F3) benchmarks

Benchmark suite	Current: `4b8243c`	Previous: `06c98bd`	Ratio
`ML-KEM-512 keypair`	`224878` cycles	`225563` cycles	`1.00`
`ML-KEM-512 encaps`	`270945` cycles	`271961` cycles	`1.00`
`ML-KEM-512 decaps`	`345296` cycles	`346678` cycles	`1.00`
`ML-KEM-768 keypair`	`373015` cycles	`374102` cycles	`1.00`
`ML-KEM-768 encaps`	`432561` cycles	`433925` cycles	`1.00`
`ML-KEM-768 decaps`	`530874` cycles	`532729` cycles	`1.00`
`ML-KEM-1024 keypair`	`562889` cycles	`562760` cycles	`1.00`
`ML-KEM-1024 encaps`	`640928` cycles	`640885` cycles	`1.00`
`ML-KEM-1024 decaps`	`762920` cycles	`762739` cycles	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

Makefile

hanno-becker

Some smaller change requests

Makefile

examples/monolithic_build/Makefile

examples/monolithic_build_multilevel/Makefile

examples/monolithic_build_native/Makefile

examples/multilevel_build/Makefile

examples/custom_backend/Makefile

scripts/tests

This commit adds the ability to measure code size for tests and examples: - For tests: run `./scripts/tests size` - For examples: run `./scripts/tests examples -l <example_name> --verbose` The function is implemented using the `size` command: - Adds the size command and corresponding Makefile configuration for both tests and each example. - Filter out empty size results to keep the output clean. - Sort results in descending order by size to enhance readability. Signed-off-by: willieyz <willie.zhao@chelpis.com>

hanno-becker

Thank you @willieyz !

mkannwischer reviewed Jun 18, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

willieyz force-pushed the measure-code-size branch from 5480b00 to 4b8243c Compare June 18, 2025 07:07

mkannwischer added the benchmark this PR should be benchmarked in CI label Jun 18, 2025

oqs-bot reviewed Jun 18, 2025

View reviewed changes

willieyz self-assigned this Jun 18, 2025

oqs-bot reviewed Jun 18, 2025

View reviewed changes

willieyz force-pushed the measure-code-size branch 5 times, most recently from 88fd0ef to f8527ad Compare June 19, 2025 11:17

mkannwischer force-pushed the measure-code-size branch from f8527ad to cdd69ad Compare June 20, 2025 07:43

hanno-becker reviewed Jul 10, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

hanno-becker requested changes Jul 10, 2025

View reviewed changes

willieyz force-pushed the measure-code-size branch from a4980d1 to 049b27c Compare July 13, 2025 18:50

willieyz requested a review from hanno-becker July 13, 2025 19:33