How Do I Evaluate a Deepfake Detection API? Ten Criteria That Matter

How do I evaluate a deepfake detection API?

Score each candidate on the same ten criteria, with evidence for each, then run a hands-on test with your own media. The first four criteria decide whether the API protects you; the rest decide whether you can run it. A strong model behind a poorly documented API with unclear error states can still cause losses in production.

Weight the criteria for your use case. A KYC onboarding flow puts heavy weight on face manipulation detection, latency and mobile SDKs. A newsroom or investigation team cares more about modality coverage, explainability and evidence export. A contact centre needs voice coverage first.

Measuring detection quality the right way

Measure on media the vendor has never seen, drawn from your own channel and passed through your own pipeline. Report MCC, the false positive rate on real users and the miss rate on each attack type at the threshold you will use. NIST AI 100-4 (November 2024) points out that accuracy, AUROC and F1 can miss what matters, because false positives on genuine content can cause serious harm.

Public benchmarks help with the shortlist. VendorBench-100, posted to arXiv in July 2026, compared 36 systems on 100 hard still images and ranked them by MCC. Neural Defend placed first with 0.876, Reality Defender second with 0.646, and Hive scored 0.290. One VendorBench author also co-authored Neural Defend's ARCAS 1B paper; the corpus and results are public, so you can rerun them. The benchmark covers still images only.

Expect your own numbers to differ from published ones. The Deepfake-Eval-2024 study, first posted in March 2025, found open-source detectors lost about 50 percent of their AUC on video, 48 percent on audio and 45 percent on images when moved from older academic benchmarks to deepfakes that circulated online in 2024. Commercial APIs vary too, which is why a side-by-side test on your media carries more weight than any brochure.

Which modalities and attack types should the API cover?

Cover the media your attackers will use in your channel, then check the attack types within each. For identity onboarding that means face images and selfie video, with face swaps, fully generated faces, AI edits and replays. For voice channels it means cloned and converted speech. For claims and evidence it means edited photos, documents and recordings.

Ask each vendor for a versioned list of supported formats and the attack types it does and does not detect. Honest gaps are useful. An API that says it does not analyse ID documents tells you to add a document control; an API that claims everything without per-type results tells you less.

Checking the response format and failure behaviour

A single fake or real label is hard to act on. Look for a calibrated or at least consistent risk score, the individual signals behind it, and enough detail for a reviewer to understand the reason for a flag. Check that every response carries a model version, so a past decision can be explained later.

Then break things on purpose. Send corrupt files, oversize files and unsupported formats; throttle the network; replay the same request twice. Confirm that each case returns a clear, documented error, that retries are safe, and that your integration never treats a missing result as a pass.

Latency, SDKs and deployment fit

Measure p95 latency from your own servers at your peak load. For a selfie check inside onboarding, the result needs to arrive well within the time a user will wait. Build a small prototype from the public documentation alone; how long that takes says a lot about the developer experience you will have later.

Match deployment options to your data rules. A managed cloud API suits most teams. Regulated buyers may need a private cloud, a secure container, on-premises or air-gapped deployment, and should confirm that the same detection quality and update path apply there.

Running a side-by-side API trial

Give every candidate API the same files, the same threshold rules, the same time limit and the same client code path. Keep the held-out set in your custody and send it only at test time. Record the model version each API reports, every error and timeout, and the raw score for each file, so you can compute metrics yourself instead of relying on vendor dashboards.

Score the results against weights agreed before the run, and keep commercial terms in a separate column so price does not quietly override a security or data requirement. Then shadow the leading API on live traffic for a few weeks before switching enforcement on.

How the NeuroVerify API measures up

NeuroVerify checks a face image, video or audio file in one API call and returns six signals: visual safety screening, live presence verification, face manipulation detection, AI generative model analysis, AI pixel-level manipulation and AI content compliance, combined into a unified risk score with signal-level evidence. Your policy sets the thresholds and actions. It does not analyse ID documents.

Eligible image requests complete in under 200 ms at p95 under defined test conditions. SDKs cover Android (Kotlin), iOS (Swift) and Flutter, plus Python, TypeScript and Node.js, Go and Java, with a REST API and an MCP server for AI agents; integration takes two lines of code and can be live in under 24 hours. Deployment runs from managed cloud to air-gapped, under ISO/IEC 27001, ISO/IEC 27701, ISO/IEC 42001 and SOC 2 Type II controls. Put it through the ten criteria above alongside any other API you are considering.