Deepfake Detection Benchmarks: How to Read Them and Which Exist

What Is a Deepfake Detection Benchmark?

A benchmark is a fixed collection of media with known labels, real or fake, plus a scoring rule. Every detector runs on the same files and the results can be compared. Some benchmarks are training datasets with a test split; others are held-out evaluation sets that no one is supposed to train on. A few are competitions with hidden test data.

A benchmark tells you how a detector performs on that collection. Whether the result carries over to your onboarding selfies, call-centre audio or newsroom submissions depends on how closely the benchmark resembles them in content, generators, compression and attack style. NIST AI 100-4 (November 2024) makes the same point: evaluations are most informative when test data reflects the content a tool will face and includes generators not seen in training.

Which Deepfake Detection Benchmarks Exist?

The table lists the benchmarks most often cited in research and vendor material, with what each actually contains. Older datasets focus on face-swap video; newer ones cover fully generated images, voice, and media collected from the open web.

How to Read Deepfake Benchmark Results

Start with the metric. Accuracy is the share of correct calls at one threshold, and it flatters any detector on an imbalanced set: if 85% of files are fake, labelling everything fake scores 85%. AUC, the area under the ROC curve, measures how well scores separate real from fake across all thresholds but does not tell you which threshold you would run in production. Equal error rate, common in voice benchmarks, is the point where false accepts equal false rejects. The Matthews correlation coefficient (MCC) uses all four cells of the confusion matrix and stays honest under imbalance, which is why VendorBench-100 uses it as its primary metric. NIST AI 100-4 highlights a fifth option: the true positive rate at a fixed low false positive rate, for workflows where wrongly flagging a real person is costly.

Then check the split. In-distribution results, where training and test fakes come from the same generators, are usually high. Cross-generator and in-the-wild results are the ones that predict production behaviour. Finally, check processing: compression, resizing, re-recording and platform transcoding all change scores, and benchmarks that skip them overstate performance.

Why Do Lab Benchmark Scores Drop in the Real World?

Because detectors learn the habits of the generators in their training data, and the world keeps producing new ones. The DFDC showed this in 2020: 2,114 participants submitted 35,109 models, the best public-leaderboard model reached 82.56%, and the overall winner scored 65.18% on a hidden black-box set built to resemble real-world conditions.

Deepfake-Eval-2024 measured the same effect on content actually circulated in 2024. Compared with earlier academic benchmarks, off-the-shelf open-source detectors lost about 50% of AUC on video, 48% on audio and 45% on images. Commercial and fine-tuned models did better, but the authors report they do not yet reach the accuracy of human deepfake forensic analysts. NIST AI 100-4 cites cross-generator image accuracy of 61% to 70% in one 2023 paper. The lesson for buyers is that a vendor's best number on FaceForensics++ says little about next year's generators.

Is There a Benchmark of Commercial Deepfake Detection APIs?

Yes. VendorBench-100 (arXiv 2607.06254, v2 dated 29 July 2026) evaluates 36 systems, including commercial APIs, zero-shot vision-language models and open-source detectors, on one adversarial corpus of 100 still images covering face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images and compressed research frames. It ranks by MCC, reports ROC-AUC, and warns that accuracy and F1 are unreliable on its imbalanced set. Version 2 corrected a scoring-normalisation error affecting the Neural Defend and TruthScan results.

Neural Defend ranked first with MCC 0.876, ROC-AUC 0.979, precision 0.963, recall 0.987 and specificity 0.857. Reality Defender was second at 0.646, Gemini third at 0.389, Hive sixth at 0.290, Sightengine eleventh at 0.258 and OpenAI's GPT twentieth at 0.135. Disclosure: one VendorBench author, Sharayu N. Deshmukh, is also a co-author of Neural Defend's ARCAS 1B paper. The corpus and results are public so anyone can rerun them. The benchmark tests still images only; it does not measure video, audio, injection attacks or production latency.

How to Benchmark Deepfake Detectors on Your Own Data

Public benchmarks shortlist vendors; your own test set picks one. Build it from the channel you need to protect, then score every candidate the same way:

• Collect genuine samples from your real traffic, with consent and data-protection approval, across devices, lighting, ages and skin tones.
• Add attacks you actually face: face swaps, fully generated faces, replayed screens, injected video and cloned voices, including generators released after the vendors' training cut-offs.
• Process samples the way your pipeline does, including compression and resizing.
• Keep a realistic ratio of genuine to fake samples and report MCC plus the true positive rate at the false positive rate your operations can absorb.
• Measure latency and failure handling under your expected load, which public image benchmarks do not cover.

Neural Defend's Published Benchmark Results

Beyond VendorBench-100, the ARCAS 1B paper (arXiv 2609.25154) reports pooled accuracy for Neural Defend's proprietary detection model of 99.99% on GenImage, 99.92% on WildFake, 99.82% on AIGC-Detect, 99.09% on AIGIBench, 97.74% on Community Forensics and 97.65% on OpenFake. These are academic image benchmarks and, as with any published result, they describe the evaluated records rather than every deployment.

In production, ARCAS 1B powers NeuroVerify, which checks a face image, video or audio with six signals in one API call and returns a unified risk score with signal-level evidence in under 200 ms p95 on eligible image requests. Teams can run it alongside an existing provider during a proof of concept and compare results on their own traffic before switching.