database: benchmark/

[hinto-janai]

Part 18

#97 <- Previous Next -> #101

Implements a standalone binary in database/benchmark/ that solely:

1. Pulls in cuprate-database
2. Tests/benchmarks all backends and various functions on the local machine
3. Prints out info in {JSON, tables, txt, etc}

Currently compiled under the workspace as target/{debug,release}/cuprate-database-benchmark[.exe].

[Boog900]

Would something like criterion be helpful instead of making something custom?

[SyntheticBird45]

Would something like criterion be helpful instead of making something custom?

Last time we wanted to test out read_from visitor pattern vs to_le_bytes, we used criterion and I find it exceptionally precise. Iirc it can also use black_box to verify implementation speed while ignoring optimization.

[hinto-janai]

Using criterion means benchmarking will be limited to cargo bench [...] instead of being a standalone binary anyone can run, right? I'm not too familiar with criterion so I'm less certain I can finish this PR in time with the feature set I had planned.

I will switch to it though, it makes more sense.

[SyntheticBird45]

Using criterion means benchmarking will be limited to cargo bench [...]

AFAIA, It uses cargo bench indeed. If providing a binary is a necessity, then criterion we should forget.

I'm not too familiar with criterion so I'm less certain I can finish this PR in time with the feature set I had planned.

I haven't followed the prs but if boog's proposal was about simplifying the work by using a crate, then I'm afraid it'll result in the opposite. If nanosecond/microsecond accuracy is the priority then criterion should be used, otherwise I suppose a custom benchmark might be quicker to build up.

[Boog900]

Is there a reason we need a standalone binary?

I'm less certain I can finish this PR in time with the feature set I had planned

I mean how much will this slow us down and what features were you thinking of? This is not necessary so I don't think we are under any rush, but at the same time if you think a standalone binary is a better fit then sure.

simplifying the work by using a crate

It was also about simplifying the code, for a standalone binary we need to add command line handling etc, whereas cargos built in bench command + criterion could handle this and a lot more.

I also plan to use criterion in other parts of the code btw.

[SyntheticBird45]

then criterion we should use

[hinto-janai]

Replaced by #324.