-
Notifications
You must be signed in to change notification settings - Fork 22
adds Dict{Symbol,DataFrame} as input for performance_profile #54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
77e2c48
874ca95
5d60df4
21cfc6d
dea02ca
0db073a
1082660
a3eac70
87064ff
1b584c6
c0e4b01
361f7ab
e9fea0e
967d883
3b6688f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -129,3 +129,45 @@ function performance_profile(b::AbstractBackend, | |
(x_plot, y_plot, max_ratio) = performance_profile_data(T, logscale=logscale, sampletol=sampletol, drawtol=drawtol) | ||
performance_profile_plot(b, x_plot, y_plot, max_ratio, xlabel, ylabel, labels, title, logscale; kwargs...) | ||
end | ||
""" | ||
performance_profile(b,stats, cost) | ||
Produce a performance profile comparing solvers in `stats` using the `cost` function. | ||
Inputs: | ||
- `b :: AbstractBackend`: the backend used to produce the plot. | ||
- `stats::Dict{Symbol,DataFrame}`: pairs of `:solver => df`; | ||
- `cost::Function`: cost function applyed to each `df`. Should return a vector with the cost of solving the problem at each row; | ||
- if the cost is zero for at least one problem, all costs will be shifted by 1; | ||
- if the solver did not solve the problem, return Inf or a negative number. | ||
Examples of cost functions: | ||
- `cost(df) = df.elapsed_time`: Simple `elapsed_time` cost. Assumes the solver solved the problem. | ||
- `cost(df) = (df.status .!= :first_order) * Inf + df.elapsed_time`: Takes the status of the solver into consideration. | ||
""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we use the same docstring format as in the existing |
||
function performance_profile(b::AbstractBackend, stats::Dict{Symbol,DataFrame}, cost, args...; kwargs...) | ||
solvers = keys(stats) | ||
dfs = (stats[s] for s in solvers) | ||
P = hcat([cost(df) for df in dfs]...) | ||
performance_profile(b, P, string.(solvers), args...; kwargs...) | ||
end | ||
|
||
performance_profile(b::AbstractBackend, T :: Array{Tn,2}, labels :: Vector{S}; kwargs...) where {Tn <: Number, S <: AbstractString} = | ||
performance_profile(b, convert(Array{Float64,2}, T), convert(Vector{AbstractString}, labels); kwargs...) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed this because it’s ambiguous. At some point I got a StackOverflow because of it. In addition, it’s not tested. |
||
|
||
""" | ||
performance_profile(b, stats, cost) | ||
Produce a performance profile comparing solvers in `stats` using the `cost` function. | ||
Inputs: | ||
- `b::AbstractBackend`: the backend used to produce the plot. | ||
- `stats::Dict{Symbol,DataFrame}`: pairs of `:solver => df`; | ||
- `cost::Function`: cost function applyed to each `df`. Should return a vector with the cost of solving the problem at each row; | ||
- if the cost is zero for at least one problem, all costs will be shifted by 1; | ||
- if the solver did not solve the problem, return Inf or a negative number. | ||
Examples of cost functions: | ||
- `cost(df) = df.elapsed_time`: Simple `elapsed_time` cost. Assumes the solver solved the problem. | ||
- `cost(df) = (df.status .!= :first_order) * Inf + df.elapsed_time`: Takes the status of the solver into consideration. | ||
""" | ||
function performance_profile(b::AbstractBackend, stats::Dict{Symbol,DataFrame}, cost, args...; kwargs...) | ||
solvers = keys(stats) | ||
dfs = (stats[s] for s in solvers) | ||
P = hcat([cost(df) for df in dfs]...) | ||
performance_profile(b, P, string.(solvers), args...; kwargs...) | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,39 +1,66 @@ | ||
using BenchmarkProfiles | ||
using DataFrames | ||
lrsantos11 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
using Test | ||
|
||
@testset "No backend" begin | ||
T = 10 * rand(25, 3) | ||
labels = ["a", "b", "c"] | ||
@test_throws ArgumentError performance_profile(PlotsBackend(), T, labels) | ||
@test_throws ArgumentError performance_profile(UnicodePlotsBackend(), T, labels) | ||
dfs = DataFrame[] | ||
for col in eachcol(T) | ||
push!(dfs, DataFrame(:perf_measure => col)) | ||
end | ||
cost(df) = df.perf_measure | ||
stats = Dict([:a, :b, :c] .=> dfs) | ||
@test_throws ArgumentError performance_profile(PlotsBackend(), stats, cost) | ||
@test_throws ArgumentError performance_profile(UnicodePlotsBackend(), stats, cost) | ||
H = rand(25, 4, 3) | ||
T = ones(10) | ||
@test_throws ArgumentError data_profile(PlotsBackend(), H, T, labels) | ||
@test_throws ArgumentError data_profile(PlotsBackend(), H, T, labels) | ||
@test_throws ArgumentError data_profile(UnicodePlotsBackend(), H, T, labels) | ||
end | ||
|
||
|
||
@testset "UnicodePlots" begin | ||
using UnicodePlots | ||
T = 10 * rand(25, 3) | ||
labels = ["a", "b", "c"] | ||
profile = performance_profile(UnicodePlotsBackend(), T, labels) | ||
@test isa(profile, UnicodePlots.Plot{BrailleCanvas}) | ||
dfs = DataFrame[] | ||
for col in eachcol(T) | ||
push!(dfs, DataFrame(:perf_measure => col)) | ||
end | ||
cost(df) = df.perf_measure | ||
stats = Dict([:a, :b, :c] .=> dfs) | ||
profile = performance_profile(UnicodePlotsBackend(), stats, cost) | ||
@test isa(profile, UnicodePlots.Plot{BrailleCanvas}) | ||
H = rand(25, 4, 3) | ||
T = ones(10) | ||
profile = data_profile(UnicodePlotsBackend(), H, T, labels) | ||
@test isa(profile, UnicodePlots.Plot{BrailleCanvas}) | ||
end | ||
|
||
|
||
if !Sys.isfreebsd() # GR_jll not available, so Plots won't install | ||
@testset "Plots" begin | ||
using Plots | ||
T = 10 * rand(25, 3) | ||
labels = ["a", "b", "c"] | ||
profile = performance_profile(PlotsBackend(), T, labels, linestyles=[:solid, :dash, :dot]) | ||
@test isa(profile, Plots.Plot) | ||
dfs = DataFrame[] | ||
for col in eachcol(T) | ||
push!(dfs, DataFrame(:perf_measure => col)) | ||
end | ||
cost(df) = df.perf_measure | ||
stats = Dict([:a, :b, :c] .=> dfs) | ||
profile = performance_profile(PlotsBackend(), stats, cost, linestyles=[:solid, :dash, :dot]) | ||
@test isa(profile, Plots.Plot) | ||
H = rand(25, 4, 3) | ||
T = ones(10) | ||
profile = data_profile(PlotsBackend(), H, T, labels) | ||
profile = data_profile(PlotsBackend(), H, T, labels, linestyles=[:solid, :dash, :dot]) | ||
@test isa(profile, Plots.Plot) | ||
end | ||
end | ||
end |
Uh oh!
There was an error while loading. Please reload this page.