Section on sensitivity analysis of ODEs is non-sensical

[mhauru]

See here: https://turinglang.org/docs/tutorials/bayesian-differential-equations/#scaling-to-large-models-adjoint-sensitivities

The text claims that the code demonstrates something about sensitivity analysis, but we just import SciMLSensitivity and then MCMC sample with NUTS.

[AoifeHughes]

Do we want this removed, clarify what is actually shown in the example, or give an actual proper example?

[penelopeysm]

I figured this out the other day (I will say that the role of each package in the DiffEq ecosystem is not very well explained). Sampling from an ODE with ForwardDiff 'just works', but to sample from an ODE with Mooncake (or reverse-mode Enzyme), it requires SciMLSensitivity to be loaded. (I think this is what they call 'adjoint sensitivities' i.e. adjoint being a keyword for reverse-mode, and sensitivity being a keyword for gradients.)

So I think the point to be made here in the docs is that:

• If you have a large model (many params), reverse mode is faster, such as Mooncake
• However, differentiating through DEs with Mooncake requires SciMLSensitivity to be loaded
• Hence we load it
• And then that lets us run the ODE example with Mooncake

[mhauru]

Sampling from an ODE with ForwardDiff 'just works', but to sample from an ODE with Mooncake (or reverse-mode Enzyme), it requires SciMLSensitivity to be loaded.

Ewww, this seems far from ideal. Is this some weird side-effect of SciMLSensitivity, or how it's intended to work and its name is just misleading? I get that the two could be related because sensitivity analysis works by taking derivatives, but this seems like a very odd interface.

[penelopeysm]

I'm not sure, SciML or AD folks would have a better understanding of it. From what I've seen I get the impression that that hooking into DEs is really complicated and can't just be solved by defining a few rules in something like MooncakeOrdinaryDiffEqExt.

[mhauru]

To summarise and add to my conversation with @AoifeHughes just now on what's needed:

1. Check you agree with Penny's understanding above, that sampling with Mooncake fails without SciMLSensitivity loaded, and works once it is loaded.
2. Maybe add a code example of how sampling with Mooncake before loading SciMLSensitivity fails, to demonstrate the issue. Although if the output is a long ugly stack trace then maybe not show that and just describe that it fails.
3. Fix the paragraph of text before the code block to explain essentially what Penny said above in her four bullet points.