How Do I Evaluate a Deepfake Detection Vendor?

How to evaluate deepfake detection vendors, step by step

A sound vendor evaluation runs in three layers. Gates decide who is eligible at all. Weighted criteria compare the eligible vendors. A residual-risk register records what no score can settle, such as a dependency on one cloud region or an untested media type. Keeping the layers separate stops a strong price or a long feature list from hiding a failed security control.

In practice that becomes six steps. Write down the decisions the detector will inform and the media involved. Set the gates: data residency, deployment mode, security certifications, contract terms and the media types you must cover. Shortlist vendors on public evidence. Run a proof of concept on a sealed test set drawn from your own channel. Score the results against weights agreed before testing. Finally, re-run the scores under different weights to check the winner does not depend on one assumption.

What should procurement compare across vendors?

Procurement should compare evidence, not feature claims. For each criterion, record what the vendor asserted, what artifact backs it, what your team observed, and the date and product version. A certification counts only if its scope covers the product and the deployment you are buying. A latency figure counts only if the test conditions are stated.

The comparison table below covers the criteria that most often change a decision. Weight them from your own loss model; a contact centre will weight voice coverage and latency far above document checks, while a lender may do the opposite.

What questions should you ask a deepfake detection vendor?

Ask questions that force specific, checkable answers. Each one below should produce a document, a number with conditions, or a test you can run yourself.

• Which public benchmarks have you been tested on, with which version, and can we rerun them?
• What are your false positive and miss rates at the threshold you recommend for our use case, and on what data?
• Which media types and attack types are covered by tested models today, as opposed to the roadmap?
• What does a reviewer see when a file is flagged, beyond a score?
• What is p95 latency for our file sizes, measured how and from where?
• Which deployment modes can you support, and what data leaves our environment in each?
• How are model updates versioned and announced, and can we pin a version?
• Which certifications cover this exact product and hosting setup?
• How do you handle files you cannot score?
• What happens to our data and integrations if we leave?

How to read public benchmarks and accuracy claims

Public benchmarks are the cheapest way to narrow a shortlist, provided you read the metric and the scope. VendorBench-100, published on arXiv in July 2026 and revised on 29 July 2026, tested 36 systems, including commercial APIs, vision-language models and open-source detectors, on one adversarial set of 100 still images and ranked them by Matthews correlation coefficient. The authors warn that accuracy and F1 mislead on this imbalanced set, because a model that labels nearly everything fake can score well on both.

The ranking shows why MCC matters. TruthScan's ROC-AUC of 0.923 is high, yet its MCC is 0.322, a sign of poor calibration at its default threshold. Neural Defend ranked first with MCC 0.876 and ROC-AUC 0.979, ahead of Reality Defender at 0.646 and 0.89. Disclosure: one VendorBench author, Sharayu N. Deshmukh, is also a co-author of Neural Defend's ARCAS 1B paper. The corpus and results are public, so anyone can rerun them. The benchmark measures still-image detection only; it does not test video, audio, injection attacks or latency.

Normalise the scores, then try to break the decision

Score every vendor at the same operating point, on the same files, with the same rounding. Do not mix results from different datasets or attack prevalence unless you adjust for it explicitly. Leave a score blank when evidence is missing rather than guessing, and never score a roadmap item as delivered.

Then recalculate under plausible changes to weights, costs and uncertain scores. ISO/IEC 23894:2023 and the NIST AI Risk Management Framework both treat risk as specific to context, which is the point: if a small change in weights flips the winner, the decision depends on weak evidence and needs more testing, not a forced rank.

Where vendor evaluation falls short

Even a careful evaluation measures the past. Attack techniques change after contracts are signed, and a vendor that leads today may lag after the next generator release. Build re-test rights, change notices and exit support into the contract so the evaluation continues after purchase. Detection also remains one control among several; FinCEN's November 2024 alert lists a detector flag as one of nine red flags, alongside signals such as webcam plugins and inconsistent device data.

How Neural Defend answers the checklist

NeuroVerify checks a face image, video or audio in one API call across six signals: Visual Safety Screening, Live Presence Verification, Face Manipulation Detection, AI Generative Model Analysis, AI Pixel-Level Manipulation and AI Content Compliance. It returns one risk score with signal-level evidence, and your policy decides. Latency is under 200 ms at p95 on eligible image requests under defined test conditions, with methodology available on request. NeuroVerify does not analyse identity documents, so pair it with your document verification provider if documents are in scope.

Deployment options are managed cloud API, private cloud or VPC, secure container, on-premises and air-gapped. Certifications are ISO/IEC 27001, ISO/IEC 27701, ISO/IEC 42001 and SOC 2 Type II. SDKs cover Android, iOS, Flutter, Python, TypeScript and Node.js, Go and Java, plus REST and an MCP server. The ARCAS 1B model behind it reports pooled accuracy from 97.65% on OpenFake to 99.99% on GenImage in the ARCAS paper. Run those claims against your own sealed set before you sign.