First Order Approximation?

[lchen64]

I wonder whether you could elaborate on the point you made in Figure 3: "We show that the first-order ODE has
the same form as their predefined forward process".

I am confused, because the marginally equivalent ODE formulation of EDM, the ODE formulation of Flow Matching and Rectified Flow, and even the ODE formulation of the continuous analog of DDPM which are depicted in the Fig3 trajectories have no higher order derivatives at all, from my understanding.

So, how does the Rectified Diffusion algorithm use the first-order approximation to the first-order reverse probability flow ODEs which is already first-order? In this sense, wouldn't it be trivial that the first-order ODE approximation has the same form as the predefined forward process, which is already first-order in nature?

Could you clarify whether I'm missing something obvious when you have time? Thanks in advance!

[G-U-N]

Hi, thank you for your comment and I just realized the ambiguity might be caused in our paper. The`first-order' as mentioned in our paper is a different concept from traditional understanding of first-order ODE.

The traditional fist-order ODE means that you don't have terms like $\frac{d^2x}{dt^2}$. So essentially, the PF-ODE of any diffusion forms are already first-order ODE.

However, the first-order as mentioned in our paper is an entirely different meaning. We hope to rectify the PF-ODE into a smoother one. Specifically,

Following the proof of DPM-Solver-1 (Equation 3.6 and the subsequent equation on Page 5 of DPM-Solver), which is equivalent to the DDIM sampler, we can approximate the ODE solution from time $t$ to $s$ as follows:

$$\mathbf{x}s = \frac{\alpha_s}{\alpha_t} \mathbf{x}t - \alpha_s \int{\lambda_t}^{\lambda_s} e^{-\lambda} \boldsymbol{\epsilon}(\mathbf{x}{t_{\lambda}}, t_{\lambda}) , \mathrm{d}\lambda + \mathcal{O}((\lambda_s - \lambda_t)^2).$$
In DPM-Solver-1, higher-order error terms $\mathcal{O}((\lambda_s - \lambda_t)^2)$ are neglected during inference, which becomes problematic when $\lambda_s - \lambda_t$ is large. \textbf{This is the primary reason why DPM-Solver-1 (DDIM) requires at least 50 steps for high-quality generation.}

In contrast, our paper does not rely on such level approximations. Our primary goal is to rectify the PF-ODE to ensure it satisfies first-order accuracy, which eliminates the high-order error terms through paired retraining (Rectification). This is the fundamental reason why Rectified Diffusion can achieve 1-step generation. In comparison, DPM-Solver requires at least several steps—typically over ten—for high-quality generation.

I am sorry for the ambiguity caused. I will revise the paper to avoid confusion.

[lchen64]

No worries!! Thank you very much for clarifying. I didn't look that carefully into Eq 3.6 and the subsequent one on Page 5 of DPM-Solver, so maybe that's why the terminology "1st order" was confusing.

But now it's clear to me exactly what your (phased) rectification model is trying to achieve. I also watched some of your talk (https://www.techbeat.net/talk-info?id=917). Interesting!

A quick followup: To your knowledge, is (Phased/Piecewise) Rectified Diffusion now SOTA (in the FID + CLIP score sense) for few step generation, since it outperforms PeRFlow on COCO 2014 and COCO 2017? (Though of course, FID is not the best measure of image quality). I'm curious whether you happened to have run experiments on CIFAR-10 already, or did you choose to only test on COCO because it is a more challenging dataset?

Anyways, it's clever to realize the need to rectify the PF-ODE to satisfy first-order accuracy and to do this through paired retraining, and the results on the COCO datasets are impressive!