benches: initial implementation #196
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hinto-janai
added
A-storage
github-actions
bot
added
A-docs
github-actions
bot
added
A-dependency
hinto-janai
changed the title
28 tasks
github-actions
bot
added
A-books
hinto-janai
changed the title
hinto-janai
commented
Oct 10, 2024
There was a problem hiding this comment.
The gist of this PR:
2 types of benchmarks exist:
- Micro benchmarks: Criterion-based
- Macro benchmarks:
cuprate-benchmarkbenchmarks
Criterion benchmarks are just another crate that uses Criterion.
cuprate-benchmark benchmarks are also kind of simple, they just implement cuprate_benchmark_lib::Benchmark and can be ran by cuprate_benchmark.
Comment on lines
5
to
42
| /// A benchmarking function and its inputs. | ||
| pub trait Benchmark { | ||
| /// The benchmark's name. | ||
| /// | ||
| /// This is automatically implemented | ||
| /// as the name of the [`Self`] type. | ||
| fn name() -> &'static str { | ||
| std::any::type_name::<Self>() | ||
| } | ||
|
|
||
| /// Input to the main benchmarking function. | ||
| /// | ||
| /// This is passed to [`Self::MAIN`]. | ||
| type Input; | ||
|
|
||
| /// Setup function to generate the input. | ||
| /// | ||
| /// This function is not timed. | ||
| const SETUP: fn() -> Self::Input; | ||
|
|
||
| /// The main function to benchmark. | ||
| /// | ||
| /// The start of the timer begins right before | ||
| /// this function is called and ends after the | ||
| /// function returns. | ||
| const MAIN: fn(Self::Input); | ||
|
|
||
| /// `cuprate-benchmark` will sleep for this [`Duration`] after | ||
| /// creating the [`Self::Input`], but before starting [`Self::MAIN`]. | ||
| /// | ||
| /// 1 second by default. | ||
| const PRE_SLEEP_DURATION: Duration = Duration::from_secs(1); | ||
|
|
||
| /// `cuprate-benchmark` will sleep for this [`Duration`] after [`Self::MAIN`]. | ||
| /// | ||
| /// 1 second by default. | ||
| const POST_SLEEP_DURATION: Duration = Duration::from_secs(1); | ||
| } |
There was a problem hiding this comment.
For cuprate-benchmark benchmarks, they are just another crate that implements cuprate_benchmark_lib::Benchmark, then...
Comment on lines
14
to
41
| /// What `main()` does: | ||
| /// 1. Run all enabled benchmarks | ||
| /// 2. Record benchmark timings | ||
| /// 3. Print timing data | ||
| /// | ||
| /// To add a new benchmark to be ran here: | ||
| /// 1. Copy + paste a `cfg_if` block | ||
| /// 2. Change it to your benchmark's feature flag | ||
| /// 3. Change it to your benchmark's type | ||
| fn main() { | ||
| log::init_logger(); | ||
|
|
||
| let mut timings = timings::Timings::new(); | ||
|
|
||
| cfg_if! { | ||
| if #[cfg(not(any(feature = "example")))] { | ||
| compile_error!("No feature specified. Use `--features $BENCHMARK_FEATURE` when building."); | ||
| } | ||
| } | ||
|
|
||
| cfg_if! { | ||
| if #[cfg(feature = "example")] { | ||
| run::run_benchmark::<cuprate_benchmark_example::Example>(&mut timings); | ||
| } | ||
| } | ||
|
|
||
| print::print_timings(&timings); | ||
| } |
There was a problem hiding this comment.
...are ran in cuprate_benchmark, which currently simply:
- Sets up inputs
- Starts a timer
- Runs benchmark
for each benchmark, prints some data. What this looks like right now:
cargo r -rp cuprate-benchmark --features example Finished `release` profile [optimized] target(s) in 0.09s
Running `target/release/cuprate-benchmark
2024-10-10T00:01:07.701083Z INFO cuprate_benchmark::log: Log level: INFO
2024-10-10T00:01:08.830698Z INFO run_benchmark: cuprate_benchmark::run: cuprate_benchmark_example::Example ... 0.12949288
Finished all benchmarks, printing results:
| Benchmark | Time (seconds) |
|------------------------------------|----------------|
| cuprate_benchmark_example::Example | 0.12949288 |
Comment on lines
+38
to
+42
| impl Benchmark for Example { | ||
| type Input = ExampleBenchmarkInput; | ||
| const SETUP: fn() -> Self::Input = example_benchmark_setup; | ||
| const MAIN: fn(Self::Input) = example_benchmark_main; | ||
| } |
There was a problem hiding this comment.
Very simple example cuprate-benchmark benchmark.
Comment on lines
+146
to
+152
| - [🟢 Benchmarking](benchmarking/intro.md) | ||
| - [🟢 Criterion](benchmarking/criterion/intro.md) | ||
| - [🟢 Creating](benchmarking/criterion/creating.md) | ||
| - [🟢 Running](benchmarking/criterion/running.md) | ||
| - [🟢 `cuprate-benchmark`](benchmarking/cuprate/intro.md) | ||
| - [🟢 Creating](benchmarking/cuprate/creating.md) | ||
| - [🟢 Running](benchmarking/cuprate/running.md) |
There was a problem hiding this comment.
More details on benchmarks are here.
This was referenced Oct 19, 2024
this `cfg()` existing makes a regular workspace `cargo b` fail
Boog900
approved these changes
Nov 20, 2024
benches/benchmark/bin/Cargo.toml
Outdated
Comment on lines
31
to
32
| cuprate-benchmark-lib = { path = "../lib" } | ||
| cuprate-benchmark-example = { path = "../example", optional = true } |
There was a problem hiding this comment.
should these be added to the workspace so we can do workspace = true
Boog900
approved these changes
Nov 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Closes #193.
This implements a very minimal
benches/:cuprate-benchmark[-lib,-example]cuprate-{example,json-rpc}criterion benchmarksWhen, Why, How, Future
See #193.
Plot examples
Here's some plots Criterion generates, taken from #322:
Improvement/regression data: