Skip to content

RebenchDB considers failed benchmarks as having runtimes of 0 seconds #224

@OctaveLarose

Description

@OctaveLarose

...which mistakenly makes benchmarks appear lightning fast, when in fact they've just failed.

See:
https://rebench.stefan-marr.de/Are-We-Fast-Yet/compare/7da4741e429375bcac6ebe2a4a5977827b599339..affb91df35f834dbc67c6e35fb71ce11ee41c965#exe-comp-somrs-interp

Data: https://rebench.stefan-marr.de/Are-We-Fast-Yet/data/5045.csv.gz

CD, DeltaBlue and Havlak fail on the BC interpreter, and are reported as having a runtime of 0 seconds.

There seems to be some sort of odd behavior in the first place where they hang and kill themselves somehow, so not a normal abort, which I will investigate. Still, RebenchDB shouldn't report them as having a 0 seconds runtime like it currently does.

Importantly, they are not present in the data file at all (as far as I can tell), so it's a RebenchDB bug and not a Rebench bug.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions