investigate whether we can specialize less

[simeonschaub]

We will likely have to be more careful than in JuliaDiff/ChainRulesTestUtils.jl#205 and JuliaDiff/ChainRules.jl#499, since there are some hot paths here we probably do want to specialize, but I suspect that there are still some gains to be had. https://timholy.github.io/SnoopCompile.jl/stable/pgdsgui/ might be helpful here.

[oxinabox]

Yeah, this code has some very hot paths, since we need to evaluate the primal function many times.