Guiding diffusion models without a classifier
Find out why classifier-free guidance is used in diffusion models and how to apply it.
Image generation may be unconditioned, or it may be conditioned on class embeddings or free-form text. When generating conditioned images, we may want to adjust how strongly the condition applies. In Diffusion Models, the guidance scale plays the same role as temperature in Large Language Models – it governs the balance between creativity and predictability. A higher guidance scale makes the model stick more closely to the caption, while a lower scale makes the resulting image more random.
Figure 1. Outputs of the stable diffusion model for the same prompt at different guidance scales. Prompt: ”yellow dog chasing a cat in the rain”, generated with huggingface.co
In Large Language Models, temperature makes it possible to flatten or sharpen the probability distribution over the tokens being sampled. Stable diffusion has no equivalent of temperature sampling. Simple approaches, such as scaling the model’s score vectors or reducing the amount of Gaussian noise added during diffusion sampling, don’t produce good results. For this reason, various methods for obtaining “low temperature” samples from a diffusion model have been put forward. The first was classifier guidance, which attached a classification network to the diffusion model to steer how strongly the output image matches a class. That method, however, doesn’t generalize to unseen classes and introduces additional training overhead. A better approach was therefore proposed – classifier-free guidance. It needs no extra network and doesn’t restrict the model to classes it has seen.
Background
At a high level, diffusion models generate an image from noise. The model progressively strips the noise from the initial noisy image until it arrives at the target high-quality image. The smaller the steps taken during denoising (i.e. the more steps), the better the resulting image.
Sampling proceeds from time T down to 0. It begins with Gaussian noise x_T and produces increasingly less noisy samples x_T −1, x_T −2, … until it reaches the final sample x_0, with t denoting the timestep. X_t can be viewed as a mix of a signal x0 and some noise, with the signal-to-noise ratio set by the timestep t.
Figure 2. A visualization of the diffusion process. Every step strips a little noise from the preceding image until a high-quality image is reached. Source: wikipedia: Diffusion_model
Training the diffusion model requires sampling an image x0, a timestep t and initial noise. From these, we can compute the noisy sample x_t at timestep t. The diffusion model learns to predict the noise between two adjacent timesteps. θ(xt, t) denotes the noise added to sample x_t at timestep t. Subtracting the predicted noise θ(xt, t) from x_t gives x_t-1. The network is trained with a mean-squared error objective between the actual noise and the predicted noise ||θ(xt, t) − θ||2.
Classifier-free guidance training
As explained above, a basic diffusion model generates unconditioned images. In other words, the generated image is completely random, and we have no control over what it will show. To condition the generated image on user input, Alex Nichol and Prafulla Dhariwal 2021 suggested adding a class embedding v_i to the timestep embedding e_t and feeding this embedding to residual blocks throughout the model. In today’s diffusion models, those residual blocks are attention layers, and the model is conditioned on text via text embeddings rather than on class embeddings.

Figure 3. How diffusion models evolved. 1. A diffusion model without class embedding. 2. A diffusion model with class embedding. 3. A classifier-guided discussion model. 4. A classifier-free diffusion model.
It became apparent that class-conditioned diffusion models have fidelity problems: the images they generate aren’t very consistent with the class. To improve constrained diffusion models further, Prafulla Dhariwal, Alex Nichol 2021 introduced classifier guidance. They placed a classification network alongside the diffusion model and used the class gradient during denoising to steer the model toward the target class. While the method performed well and exchanged diversity for fidelity, it came with drawbacks. The biggest one was that the model couldn’t generalize to classes absent from training. Another issue was that the classifier needed to be trained on noisy samples, which ruled out plugging in a pre-trained general-purpose classifier.

Figure 4. Fréchet inception distance/Inception score plotted against guidance strength. Every curve corresponds to a model with unconditional training probability p_uncond, and every point to a different guidance scale. Stronger guidance leads to a higher FID/IS score. A lower FID score is better, and a higher IS score is better. The plot makes it clear that guidance strength trades FID for IS. FID penalizes a lack of variety. IS, by contrast, measures how well a model captures the entire ImageNet class distribution while generating individual samples with low entropy for a single class. A weakness of this metric is that it doesn’t reward covering the whole distribution or capturing diversity within a class, so models that memorize a small part of the full dataset still achieve a high IS. Source: arxiv.org
To address these problems, Jonathan Ho & Tim Salimans introduced classifier-free diffusion guidance. They demonstrate that training a conditional and an unconditional diffusion model together and combining the resulting conditional and unconditional scores makes it possible to trade off sample quality against diversity. Specifically, the noise at timestep t θ(xt, t, c) is trained both with and without the class condition. In the unconditioned model, the class identifier c is replaced by a null token ∅ (θ(xt, t, c = ∅)). Both the conditioned and unconditioned models can be trained together by randomly setting c to ∅. In classifier-free guidance, the inference parameter w determines how strong the condition is:
θ_overall(xt, t, c) = (1 + w)θ(zt, t, c) − wθ(xt, t)
One way to read the equation is that the diversity in the image is subtracted from the distribution of a given class, which reinforces that particular class distribution even more.
Figure 5. How guidance strength affects a mixture of three Gaussians, each standing for a different class. Without guidance, the classes overlap to some extent, so sampling from one class can yield samples belonging to other classes. Raising the guidance strength separates the classes more clearly, which increases fidelity. Source: arxiv.org
Summary
In this article, we explained why classifier-free guidance is used in diffusion models and how it works. When generating images, we want control over what gets generated, which calls for conditioned diffusion models. Simple conditioned diffusion models, however, lack fidelity and class consistency, so some diversity has to be given up in exchange for fidelity. That’s why classifier guidance was proposed first, followed by the improved classifier-free guidance, as ways of controlling diversity versus fidelity. Diffusion models can also be conditioned on other signals, such as other images. This form of diffusion control, though, relies on different mechanisms: control images act directly on convolutional layers, so neither classifier nor classifier-free diffusion guidance is needed. And if diffusion models interest you, the blogposts on diffusion model evaluation metrics and on speeding up diffusion model training with min-SNR may also be worth reading.