As explanation, something I wrote previously:The most common approach to modeling continuous distributions is to train a reversible model f that maps it to another continuous distribution P that is already known. The original image can be recovered by tracking the bits needed to encode its latent, as well as the reverse path:<pre><code> −log P(f(x)) − log|det ∂f&#x2F;∂x (x)|
</code></pre>
This technique is known as normalizing flows, as usually a normal distribution is chosen for the known distribution. The second term can be a little hard to compute, so diffusion models approximate it by using a stochastic PDE for the mapping. When f is a solution to an ordinary differential equation,<pre><code> dx&#x2F;dt = g(x)
</code></pre>
then<pre><code> log|det ∂f&#x2F;∂x (x)| = ∫ Tr(∂g(x)&#x2F;∂x) dt = ∫ E_{ε∼N(0,I)} [εᵀ ∂g(x)&#x2F;∂x ε] dt
</code></pre>
The last equality is known as Hutchison&#x27;s estimator. Switching to a stochastic PDE<pre><code> dx′ = g(x′)dt + ε(t)dW
</code></pre>
and tracking the difference δx = x′ − x, the mean-squared error approximately satisfies<pre><code> d(δxᵀδx)&#x2F;dt = 2δxᵀ ∂g(x)&#x2F;∂x δx,
</code></pre>
which is close to Hutchinson&#x27;s estimator, but weighted a little strange.

I briefly covered that connection in an earlier blog post: <a href="https:&#x2F;&#x2F;sander.ai&#x2F;2023&#x2F;07&#x2F;20&#x2F;perspectives.html#flow" rel="nofollow">https:&#x2F;&#x2F;sander.ai&#x2F;2023&#x2F;07&#x2F;20&#x2F;perspectives.html#flow</a>
... but it&#x27;s definitely something that might deserve a longer-form treatment at some point :)

It is a good post, but is missing the connection to continuous normalizing flows. Diffusion models, flow matching, consistency models are biased approximations of continuous normalizing flows (which themselves have some slight biases, but less). Adversarial losses can somewhat help with bias (e.g. RL, GANs), but training those has issues.

MIT&#x27;s OCW is usually pretty reliable:<a href="https:&#x2F;&#x2F;www.practical-diffusion.org&#x2F;lectures&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.practical-diffusion.org&#x2F;lectures&#x2F;</a>There is more math-heavy <a href="https:&#x2F;&#x2F;diffusion.csail.mit.edu&#x2F;2026&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;diffusion.csail.mit.edu&#x2F;2026&#x2F;index.html</a>

I really liked Calvin Lou’s tutorial. It’s quite old at this point and probably not up to date, but I felt it was awesome for understanding the concepts

Does anyone have good resources into a more practical approach toward building diffusion models? I found the book by Rashka for Building an LLM from Scratch really helpful in understanding a lot of concepts behind LLMs, and I am looking for a similar resource for diffusion models

Haven&#x27;t finished this but for me it&#x27;s so refreshing to read some science on deep learning and not just weird predictions.

- just a headsup- your links to the slides for deeplearning you did here <a href="https:&#x2F;&#x2F;sander.ai&#x2F;2014&#x2F;05&#x2F;29&#x2F;slides-meetup.html" rel="nofollow">https:&#x2F;&#x2F;sander.ai&#x2F;2014&#x2F;05&#x2F;29&#x2F;slides-meetup.html</a> are broken

Very helpful! Naïve question (I haven’t had a chance to read TFA at all and diffusion&#x2F;flow models are not my area of expertise). Doesn’t learning the integral&#x2F;solution of the diffusion process in a single pass just take us back to like OG generative CNN that we had before diffusion models took over? Surely the answer is “no” but would love to hear your framing as to why.

Why is self-distillation necessary? Why can&#x27;t they get the ground-truth for &quot;skipping&quot; steps?

Diffusion and flow matching models generate samples by iterative denoising. Iterative denoising means passing input to the neural network, running a forward pass, and taking the output back as input and rerunning the neural network. Often you do this 100 times, which is slow and expensive.Flow maps &#x2F; consistency models &#x2F; shortcut models instead try to learn to compress this iterative work into 1 forward pass. This makes inference 100x faster as you&#x27;d only need to run the neural net forward pass once. Beyond speeding up inference, there are other advanced benefits to this, such as improved ability to perform inference-time steering.Mathematically, learning a flow map corresponds to learning to solve an ordinary differential equation, i.e., learning the time integral of the velocity field. This mathematical foundation provides the basis for various training objectives for learning flow maps, which involve self-referential identities or identities such as the transport equation, which are discussed in the blog post.Hope that helps! I&#x27;m an ML researcher currently researching flow maps.

This provides a high-level overview of diffusion models, you know, the models behind Stable Diffusion, Gemini banana, etc.I haven&#x27;t read it carefully but I think it&#x27;s pretty comprehensive. From SDE to Flow matching formulation, and different perspective of constructing the flow maps, i.e. x-formulation or v-formulation. It also deals with distillation and consistency, which is used to fast sampling.Overall, it&#x27;s a good read if you are new to the field.

HN is a place where it&#x27;s legitimate to ask those kinds of questions. The site has a high concentration of advanced practitioners -- in my experience it is not uncommon for the creator of a technology or deep expert to reply. John Carmack has an account on the site for instance. :)

We&#x27;ve all seen that AI can give you plausible but incorrect answers. Having an expert read it or use AI on it and interpret and validate it before posting would be most welcome IMO.

Why not put it into an AI yourself? :) I&#x27;d rather we avoided a precedent of asking for it and N people replying with their own favorite AI version. The comments section would end up a ghost town.Extreme TL;DR: Diffusion models are like getting f(x) by calculating and summing f&#x27;(0), f&#x27;(1)...f&#x27;(x). Flow models are like just calculating f(x).

This is way outside of my expertise, can anyone given a TL;DR or ai;dr?

Learning the Integral of a Diffusion Model