What Separates a Deepfake Demo From an Operational Attack?

What separates a deepfake demo from an operational attack?

A deepfake demo answers a narrow question: can this model produce a convincing face, voice or video for a chosen input on stated hardware? An operational attack answers a different one: did the fake change what a real person or system did? The demo ends when the output looks right. The attack only starts there.

Research papers usually report selected examples, quality metrics on a fixed test set and throughput on a specific GPU. Those are honest results for the setup described. They do not measure how easy the tool is for criminals to obtain, how much reference material a target leaves online, how the media survives a messaging app, or whether a finance clerk would act on it.

The reverse also holds. An attacker does not need research-grade output. The FBI Internet Crime Complaint Center reported in December 2024 that criminals use generated images for fake profiles and forged IDs, short cloned voice clips for fake emergencies and bank account access, and real-time video calls with fake executives or police. Many of those media files would look weak in a lab comparison and still worked because the surrounding story was believable.

Why does deepfake detection accuracy drop in the real world compared with lab benchmarks?

Detection accuracy drops in the real world because most academic benchmarks were built from a small number of generators, clean source video and consistent compression, and detectors learn those habits. Field media breaks all three. It comes from newer generators, it has been resized and recompressed several times, and it is mixed with heavily edited genuine content that can look just as unusual.

The clearest public measurement is Deepfake-Eval-2024, released on arXiv in March 2025 by Chandra and colleagues. The team collected deepfakes that circulated in 2024 from social media and from users of a detection platform: about 45 hours of video, 56.5 hours of audio and 1,975 images from 88 websites in 52 languages. Open-source detectors that scored well on older benchmarks lost roughly half their AUC on this set. Commercial tools and models fine-tuned on the data did better, but the authors report that none matched trained human forensic analysts.

Three mechanisms account for most of the gap. Generator drift means the detector meets artifacts it was never trained on. Transformation loss means the high-frequency traces many detectors rely on are flattened by compression, scaling and re-recording. Base-rate shift means fakes are rare in production traffic, so a false positive rate that looked tiny on a balanced test set produces a queue full of genuine users.

The five stages an operational deepfake attack has to clear

Treating an attack as a chain makes it clear where a defender can break it. Each stage has its own failure points, and the media file matters in only some of them.

Acquisition comes first: the attacker needs usable photos, video or voice of the person being imitated, plus enough background to build a pretext. Preparation follows, where the attacker generates the media and often edits it by hand to fix obvious errors. Delivery is where quality usually suffers, because phone networks, chat apps and video platforms compress what they carry. Exploitation is the conversation or upload itself, where urgency and authority do most of the work. Cash-out is the payment, credential reset or account opening that the whole effort was for.

Public incident reporting rarely covers every stage. In May 2025 the FBI warned that actors were sending AI-generated voice messages while impersonating senior US officials, but the alert did not name the tools or publish samples. That is normal for advisories, and it means a page like this one should not guess which generator was used. The tactic is documented; the toolchain is not.

• Acquisition: limit public reference material for high-risk staff and watch for scraping of executive media.
• Delivery: inspect media at the point it enters your system, before downstream steps strip metadata.
• Exploitation: require a callback or second approver for any request that moves money or access.
• Cash-out: hold or review first-time payees, new devices and recovery-factor changes.

Which evidence answers which question

Analysts often merge capability, prevalence and control effectiveness into one claim. Keeping them apart produces a sounder risk assessment. A paper can show that real-time face reenactment runs at video frame rates. An advisory can show that criminals use live video impersonation. Only your own exercise can show whether your help desk would catch it.

NIST AI 100-4, published in November 2024, makes a similar point about synthetic content: detection, provenance and labelling each address part of the problem, and their performance depends on the setting in which they are used. When one link in the evidence chain is missing, say so in the assessment rather than filling it with the most likely tool name.

How to test detection against operational conditions

Build the evaluation around the decision you are protecting. Collect genuine samples from your own capture channel, then create fakes with several current generators and push them through the same path a real attacker would use: the mobile upload, the contact-centre codec, the meeting platform. A guide to representative test sets covers how to size and label that data.

Apply the transformations attackers get for free. Recompress, resize, crop, screen-record and add background noise, then measure how much the score moves. The guide to transformation testing lists the common ones. Report results with MCC, false positive rate at your planned threshold and the share of samples the system declined to score, since accuracy alone hides most of what matters on imbalanced data.

Finally, run the scenario end to end at least once a quarter. A detector that flags a test video is only useful if the alert reaches someone with the authority to stop the payment or reset.

How NeuroDetect handles media from the field

NeuroDetect is Neural Defend's forensic workspace for authorized images, audio, video and documents. An analyst opens a case, uploads the sample and gets detailed views down to regions, frames and audio segments, including explainability heatmaps and attention views. Each result keeps the model and version that produced it, which matters when a case is reopened months later and the models have changed.

The underlying model, ARCAS 1B, was evaluated across several public benchmark families. The ARCAS paper reports pooled accuracy of 99.99% on GenImage, 99.92% on WildFake, 97.74% on Community Forensics and 97.65% on OpenFake. The authors state that those results describe only the records evaluated, which is the honest way to read any benchmark after looking at Deepfake-Eval-2024.

For a harder test, the public VendorBench-100 benchmark scores 36 systems on 100 adversarial still images chosen as edge cases: face swaps, AI photo edits, avatar compositing and compressed research frames. Neural Defend ranked first with an MCC of 0.876, against 0.646 for the next vendor. One VendorBench author also co-authored the ARCAS paper; the corpus and code are public so anyone can rerun the scoring. VendorBench covers still images only, not video or audio.

The downloadable forensic report records observations, uncertainty, provenance and analyst notes, so a field sample that scores in an ambiguous range is documented as ambiguous rather than forced into a verdict. The analyst makes the final call.