How a video deepfake detection pipeline works, stage by stage
Video detection is image detection plus time, plus sound. The pipeline starts with a bounded probe of the file: container, codecs, duration, frame rate, dimensions and the list of video, audio and subtitle streams. It then decodes and samples frames, detects faces, follows each face across frames as a track, and runs models on crops and full frames. A temporal stage looks for flicker, unstable boundaries and motion that real faces do not produce. An audio stage scores the speech track, and an audio-visual stage can compare lip movement with the sound. The last stage aggregates everything into results a reviewer can act on.
Each stage can fail on its own. A variable frame rate can skew timestamps, a corrupted stream can decode partially, and a face detector can lose a track when the head turns. Record those events with the result, because a clean-looking verdict built on half the footage is misleading.
- Probe and validate: format, size, streams, decode warnings.
- Sample: fixed interval, scene change, motion or face-track driven.
- Detect and track faces: one track per subject, with coordinates per frame.
- Score frames: spatial artefacts, blending boundaries, generator fingerprints.
- Score time and sound: temporal consistency, speech synthesis traces, lip-sync alignment.
- Aggregate and report: per-modality results, analysed intervals, unresolved regions.
How many frames should a video deepfake detector analyse?
Enough to cover the risk, and no more than the latency budget allows. Full-frame analysis of a 30 frames-per-second, 10-minute video means 18,000 frames, which is rarely practical in a live workflow. Most systems sample. Neural Defend's public video API, for example, samples frames uniformly, 12 by default, and lets the caller set max_frames from 1 to 100 or override the sample rate.
The right number depends on what you are checking. For a 10-second onboarding liveness clip, 12 frames gives a sample roughly every second, which suits a single continuous face. For a long recorded interview where only one answer may be manipulated, sparse uniform sampling can miss the segment entirely, so raise the frame count, sample by scene change, or analyse the flagged section again at higher density.
Whatever you choose, store the parameter with the result. Two analyses of the same file with different sampling can disagree, and an investigator needs to know why.
| Strategy | How it works | Good for | What it can miss |
|---|---|---|---|
| Uniform interval | Evenly spaced frames across the file | Short clips with one subject | Brief manipulated segments between samples |
| Scene change | Frames at cuts and shot boundaries | Edited footage, news clips | Manipulation inside a long static shot |
| Face-track driven | Frames chosen per tracked face | Interviews, calls, multi-person video | Background or body edits |
| Dense re-scan | Every frame in a flagged interval | Investigation of a suspect section | Nothing in range, but costly |
What temporal and audio analysis add over frame scoring
Single frames from modern generators can look clean. Problems show up over time: a jawline that shimmers between frames, skin texture that resets after a head turn, blinking that follows an unnatural rhythm, or a face boundary that slides against the hair. Temporal models compare neighbouring frames along a face track to catch these.
Audio carries its own evidence. A cloned voice can be scored for synthesis traces even if the video is genuine, and a real voice can be dubbed over a face-swapped video. Lip-sync checks compare mouth shapes with phonemes; drift between them is a sign that one of the two streams was generated or edited. Because these failure modes are independent, NeuroVerify returns separate video and audio risk fields rather than one blended number, and leaves any combined decision to your documented policy.
Aggregation and coverage: what the video result must include
Aggregation turns many frame and segment scores into a result. Taking the maximum is sensitive but noisy; taking the mean can dilute a short manipulated segment into a low score. Many systems use a high percentile or require several consecutive suspicious frames. Whichever rule you use, document it and keep the per-frame scores, so a reviewer can trace the summary back to timestamps and face tracks.
A defensible video result records the aggregation rule, the frames and intervals analysed, which faces were tracked, which frames were skipped and why, and which streams had no score. NIST's November 2024 report on synthetic content (NIST AI 100-4) treats detection as one of several transparency techniques with known limits, which is another reason to show coverage instead of a bare verdict.
Running video analysis in production
Video jobs vary in length, so plan for queues. Separate upload completion, probing, decoding, scoring and reporting into states you can observe. Track accepted duration against analysed duration, queue age, partial jobs and stage failures, and split those metrics by model version. A drop in analysed duration often signals a decoder or codec problem long before anyone notices a change in detection rates.
File limits shape the intake design. NeuroVerify's video endpoint accepts MP4, AVI, MOV, MKV, WMV, FLV, WebM, OGG and OGV up to 1.5 GB. For longer recordings, split by scene or time window, analyse each part, and keep the mapping back to the original timeline.
Where video detection helps and where it falls short
Video detection is strong at finding face swaps, reenactment and generated faces in recorded media, and at pointing a reviewer to the exact frames that look wrong. It is weaker when footage is heavily compressed, very low resolution, or re-recorded from a screen, because those processes erase the fine artefacts models rely on. It cannot tell you whether genuine footage is being presented out of context. The authors of the DeepFake Detection Challenge dataset paper (Dolhansky and colleagues, 2020) described video deepfake detection as extremely difficult and unsolved, and the gap between lab benchmarks and messy real footage has not closed since.
For live calls and onboarding streams, the larger risk can be injection: a deepfake fed through a virtual camera so the platform receives synthetic frames as if from a lens. Pair video analysis with capture-side and presence checks, and treat every score as input to a decision a person or policy owns.
How Neural Defend analyses video
Neural Defend gives two entry points. NeuroVerify's video API returns separate video and audio risk scores for workflow decisions such as onboarding and account recovery, alongside the six signals it runs on face media. NeuroDetect is the forensic workspace for investigators: upload authorized video into a case, switch between the original, explainability heatmaps and attention views down to individual frames, regions and audio segments, read an LLM-assisted summary linked to that evidence, and download a forensic report with observations, uncertainty, provenance and model context. The analyst makes the final call.
Both run on ARCAS 1B, Neural Defend's own detection model. Its paper reports pooled accuracy in Table 1 of 99.99% on GenImage and 97.65% on OpenFake, among other datasets; those are image benchmarks, so test video performance on your own footage. Deployment options include private cloud, on-premises and air-gapped installs for evidence that cannot leave your boundary.
Evidence boundary
Limitations
- ARCAS 1B accuracy figures come from image datasets reported in the ARCAS paper and do not describe video performance on your footage.
- NeuroVerify video details reflect Neural Defend's public developer documentation as of August 2026.
- Heavily compressed, low-resolution or screen-recorded video reduces the artefacts any detector can use.
- Video detection does not establish whether genuine footage is being used out of context.
Source record
Sources and references
- National Institute of Standards and TechnologyReducing Risks Posed by Synthetic Content: An Overview of Technical Approaches to Digital Content Transparency (NIST AI 100-4)Published November 20, 2024. Verified October 19, 2026.
- arXivThe DeepFake Detection Challenge (DFDC) DatasetPublished June 12, 2020. Verified October 19, 2026.
- Neural DefendVideo deepfake detection APIVerified October 19, 2026.
- Neural DefendNeuroDetect forensic media workspaceVerified October 19, 2026.
- arXivBenchmarking Neural Defend ARCAS 1B: A Foundational Multimodal Deepfake Detection ModelPublished September 21, 2026. Verified October 19, 2026.
