How Do Diffusion Models Generate Images?

How do diffusion models generate images?

A diffusion model generates an image by reversing a noising process. Training takes a real image, adds Gaussian noise in a fixed schedule until only static is left, and teaches a neural network to predict the noise that was added at each level. Generation runs the process backwards: the model begins with pure noise and, over dozens to hundreds of steps, subtracts its own noise estimate until a clean image remains.

Nothing in the network stores a training picture to be retrieved. Each run starts from a fresh noise sample, so the same prompt yields a different image every time. That is why diffusion tools produce endless variations, and why a suspect image usually has no single original to find.

The same mechanism works for other media. Audio models denoise spectrograms or waveforms, and video models denoise a stack of frames together so that motion stays consistent. Most image products now add two components around the core loop: a text encoder that turns the prompt into numbers, and an autoencoder that moves between pixels and a smaller latent space.

How did diffusion models change image generation in 2020?

Diffusion changed image generation in 2020 by showing that a denoising model could match or beat GANs on image quality without adversarial training. The idea had appeared in 2015, but the Denoising Diffusion Probabilistic Models paper by Jonathan Ho, Ajay Jain and Pieter Abbeel, posted on 19 June 2020, made it work at scale. It reported an Inception score of 9.46 and an FID of 3.17 on unconditional CIFAR-10, and sample quality on 256-pixel LSUN images comparable to ProgressiveGAN.

GANs pit a generator against a discriminator and are known for unstable training and mode collapse, where the generator keeps producing a narrow range of outputs. DDPM avoided that contest entirely. The cost moved to inference: generating one image required hundreds of network evaluations instead of one forward pass. Later samplers such as DDIM reduced the step count, but the 2020 paper settled the direction research took.

DDPM itself had no text prompts, no identity control and no video. Those capabilities arrived over the next two years, and reading them back into 2020 distorts the timeline.

How did text-to-image generation move from DALL-E to latent diffusion?

Text-to-image generation moved from token prediction to conditioned denoising between January 2021 and 2022. OpenAI announced DALL-E and CLIP on 5 January 2021. DALL-E treated an image as a sequence of tokens and predicted them from the text, much like a language model writes words. CLIP scored each candidate image against the prompt, and the best-matching samples were shown, so the published examples reflected generation plus selection.

Latent diffusion took a different route. Instead of predicting tokens, it denoised a compressed image representation and injected the prompt into every step through cross-attention layers. The same framework handled text-to-image, inpainting, super-resolution and layout-to-image tasks. DALL-E and latent diffusion are separate architectures, not successive versions of one design.

For anyone checking images, this shift matters because prompt-driven diffusion made it trivial to ask for a specific scene, a style or a person-like face, then regenerate until the result looked right. A realistic photo can still get the prompt, anatomy or facts wrong, which is one reason visual plausibility and authenticity have to be judged separately.

Why did latent diffusion make text-to-image models easier to run?

Latent diffusion made text-to-image models easier to run because it denoises a small compressed version of the image rather than every pixel. Robin Rombach and colleagues, in a paper posted on 20 December 2021 and presented at CVPR 2022, trained a pretrained autoencoder to shrink images and ran diffusion in that latent space. They noted that pixel-space diffusion models often needed hundreds of GPU days to train and were expensive to sample, and that the latent approach cut those requirements while keeping visual detail.

Release decisions did the rest. Stability AI made Stable Diffusion, built on this approach, publicly available on 22 August 2022 under the CreativeML OpenRAIL-M licence, saying it needed about 6.9 GB of GPU memory. Downloadable weights meant anyone with a mid-range graphics card could generate images offline, fine-tune the model on new faces or styles and share the modified checkpoints.

Lower cost and open weights explain why diffusion images spread so quickly. They do not show who used a model in a particular incident, whether they followed the licence, or which checkpoint produced a given file.

Why diffusion fakes slip past detectors built for GANs

Many early deepfake detectors learned the fingerprints of GAN upsampling, such as periodic patterns in the frequency spectrum. Diffusion images are produced differently and often lack those patterns, so a detector trained mostly on GAN data can score them as real. The problem is worse for fine-tuned community checkpoints, which shift textures away from any base model a detector has seen.

Diffusion also complicates attribution. A single delivered image may combine a base model, a sampler, guidance settings, a reference adapter, an inpainting mask, a decoder and a round of manual editing. A detector can say an image shows signs of diffusion generation; naming the exact model and version requires a system built and validated for that task, and most are not.

• Test detectors on several diffusion families and on fine-tuned checkpoints, not one popular model.
• Include inpainted images where only a region was generated, since whole-image scores can miss local edits.
• Recompress and resize test images the way social platforms do before measuring results.
• Keep the original received file; screenshots and re-saves destroy the faint traces detectors rely on.

How NeuroDetect examines suspected diffusion images

NeuroDetect, Neural Defend's forensic workspace, lets an analyst upload an authorized image into a case and review it at region level, with explainability heatmaps and attention views showing which areas drove the result. That matters for diffusion fakes, where an inpainted face can sit inside an otherwise genuine photo. The case keeps the model and version used, so the finding can be re-examined later.

The detection model behind it, ARCAS 1B, was evaluated on generator-diverse image benchmarks. The ARCAS paper reports pooled accuracy of 99.99% on GenImage, which spans several diffusion and GAN generators, 99.92% on WildFake and 99.09% on AIGIBench. On the public VendorBench-100 set of 100 adversarial still images, which includes AI photo edits and text-to-video stills, Neural Defend ranked first of 36 systems with an MCC of 0.876. One VendorBench author also co-authored the ARCAS paper, and the corpus is public for anyone to rerun.

For high-volume screening of selfies and profile photos rather than case work, NeuroVerify runs the same kind of analysis through one API call, including AI Generative Model Analysis and AI Pixel-Level Manipulation signals.