How Are Deepfakes Made?

How do deepfakes work?

Deepfakes work by teaching a neural network the statistical patterns of faces or voices, then using it to produce new frames or audio that follow those patterns while matching a chosen identity, expression or script. The network does not copy and paste. It infers what the target would look or sound like in a situation that was never recorded.

Two ingredients make this possible. The first is a learned representation, a compact numerical description of identity, expression, pose, lighting or voice that the model can adjust independently. The second is a generator that turns those numbers back into pixels or sound. Different model families, including autoencoders, GANs and diffusion models, handle these two steps in different ways.

Production rarely ends at the model output. Creators select the best attempts, fix errors by hand, blend the generated region into real footage, add background noise and compress the result. That post-processing affects both how convincing the fake is and which traces survive for detectors.

How are deepfakes made? The five main methods

Deepfakes are made with one or more of five methods, chosen according to what the creator wants to change. A face swap replaces one person's face in existing video with another's. The classic approach trains an autoencoder with one shared encoder and a separate decoder per person; passing person A's frames through person B's decoder renders B's face with A's expressions.

Reenactment keeps the target's identity and drives their face with someone else's expressions or head movements, as Face2Face showed in real time in 2016. Lip sync regenerates only the mouth region to match new audio. Full synthesis generates an entire face, scene or video from noise or a text prompt using GANs or diffusion models. Voice cloning produces speech in a target's voice from a sample, either from text or by converting another speaker's voice.

Attacks often stack methods. A fake executive on a video call might combine a real-time face swap with a cloned voice. A synthetic identity might pair a fully generated face on a forged document with a face-swapped selfie video.

How much data does a deepfake need?

The data a deepfake needs has fallen from hours to seconds for some methods. Vice's Motherboard reported in December 2017 that the first Reddit face swaps were trained on collections of images of each person gathered from image search, stock photos and YouTube videos. Synthesizing Obama in 2017 used many hours of weekly address footage of one speaker.

Zero-shot models changed that. Microsoft's VALL-E paper, posted on 5 January 2023, synthesised speech in an unseen speaker's voice from a 3-second recording used as a prompt. Wav2Lip (2020) syncs any face without training on that person. Diffusion models can produce a convincing synthetic face from a sentence, and reference-image adapters can make the output resemble a real person from one or a few photos.

Quality still improves with more data. A short sample produces a recognisable voice; a longer, clean sample captures accent, rhythm and emotion more convincingly. Executives, creators and public officials with hours of public video are the easiest targets for high-quality fakes.

What software is used to make deepfakes?

Software used to make deepfakes falls into four broad groups. Open-source face-swap frameworks, such as DeepFaceLab, which its authors described in a 2020 paper as the dominant face-swapping framework, package the autoencoder method with tools for extracting, training and merging faces. Released research models and weights, such as Wav2Lip for lip sync and Stable Diffusion for images, let anyone with a capable GPU run the methods locally and fine-tune them.

Hosted generators, including text-to-video services such as OpenAI's Sora and Google's Veo, produce realistic clips from prompts with no setup, though they apply usage policies and provenance marks such as watermarks or C2PA metadata. Real-time tools route a live face swap or voice conversion into a virtual camera or microphone so it appears in a video call or a remote identity check.

Most legitimate users of these tools are filmmakers, dubbing studios, game developers and researchers. Misuse comes from how the output is presented: as a real person, without consent, to obtain money, access or attention.

• Open-source face-swap frameworks: offline video face replacement.
• Released model weights: local image, lip-sync and voice generation that can be fine-tuned.
• Hosted generators: prompt-based images, video and voices with vendor policies and marks.
• Real-time camera and voice tools: live manipulation in calls and verification sessions.

How production methods shape detection

Each method leaves different evidence, so no single check catches all of them. A swap may show a boundary around the face; a lip sync shows problems only in the mouth; a fully synthetic image has no boundary at all but carries statistical traces of the generator; a cloned voice needs audio analysis. A detector trained on one family can miss another, and compression or screen recording removes some evidence entirely.

Delivery path matters as much as method. A fake uploaded as a file can be compared against provenance data. A fake streamed through a virtual camera into a KYC session needs capture-integrity checks as well as media analysis. A cloned voice on a phone call has passed through narrowband codecs that blur many artifacts.

How Neural Defend checks for each method

Neural Defend's products analyse the outputs of these methods rather than relying on one artifact type. NeuroDetect, the forensic workspace, shows investigators evidence down to regions, frames and audio segments, so an analyst can see whether a finding points to a swapped face, a regenerated mouth or a fully generated image. Each result keeps the model and version used.

For identity flows, NeuroVerify runs six signals in one API call, including Face Manipulation Detection for swaps and reenactment, AI Generative Model Analysis for synthetic faces, AI Pixel-Level Manipulation and Live Presence Verification, and returns one risk score with signal-level evidence in under 200 ms p95 on eligible image requests. On the public VendorBench-100 benchmark of 100 adversarial still images covering face swaps, AI photo edits and avatar compositing, Neural Defend ranked first of 36 systems with an MCC of 0.876. One VendorBench author also co-authored Neural Defend's ARCAS paper, and the corpus is public.