Denoising Diffusion Probabilistic Model Explained in High-Level Overviews

Simplify the explanation of generative modeling and denoising diffusion probabilistic models

Generative Modeling Growth

• In the past five years, generative modeling has made significant advances.
• Models like VAEs, GANs, and flow-based models have been successful in creating high-quality images.
• Diffusion models are a newer type that outperforms previous methods.

Diffusion Models Explained

• These models are based on principles from non-equilibrium thermodynamics.
• They learn to create images through a process called denoising.

Two-Step Process

1. Forward Process:

In the forward process, we gradually add random noise to the data over a series of time steps (X0, X1, …, X_T).

• At each time step, we draw samples from a Gaussian distribution.
• The mean of this distribution is conditioned on the sample at the previous time step.
• The variance of the distribution follows a fixed schedule.
• By the end of the forward process, the samples end up with a pure noise distribution.

Modified from source

2. Reverse Process:

During the reverse process, we aim to undo the added noise at each time step (X_T, X_T-1, …, X0), aiming to recover the original data.

• Start with the pure noise distribution (which corresponds to the last step of the forward process).
• Then, denoise the samples backward from X_T to X0.

Modified from source

Denoising Diffusion Probabilistic Models (DDPMs)

• The DDPM paper introduced diffusion models for creating high-quality images.
• It showed that diffusion models could be equivalent to other complex training methods, resulting in excellent images.

Training and Sampling

The paper presents two algorithms:

• One for training the model by optimizing a specific mathematical function.
• Another for sampling from the trained model to create new images.

DDPM

Note on DDPM

• DDPM is one approach to diffusion models but can be slow in generating new samples.
• Research like Denoising Diffusion Implicit Models (DDIM) aims to speed up this process.

Implementing DDPM

To implement a DDPM, we create a model that takes images and time steps as inputs.

During training, we:

1. Add random noise to the images.
2. Use the model to predict and remove the noise.
3. Update the model based on how well it removes the noise.

Conclusion

By teaching the model to remove noise, we can use it to generate new images from pure noise. This is a simplified explanation, but it captures the essence of how diffusion models work in generative modeling. 🌟