Multimodal Deepfake Detection Architecture: Fusing Voice, Video, Image and Document Evidence

What is multimodal deepfake detection and is it better?

Multimodal deepfake detection analyses more than one kind of signal from the same event, such as the face and the voice in a video call, or the selfie and the uploaded document in an onboarding, and combines the findings. It is better in a specific sense: it can catch attacks where each channel looks plausible alone but the combination does not, and it degrades more gracefully when one channel is low quality.

It is not automatically more accurate. Adding a weak analyzer can add noise, and a fused score can hide the one modality that actually detected the attack. The researchers behind FakeAVCeleb, a 2021 audio-video deepfake dataset, argued that detectors need to treat audio and video together because attackers now synthesise both. AV-Deepfake1M, published in 2023 with more than 1 million videos of over 2,000 subjects, added content-driven manipulations of short segments inside real videos and found that localising them remained difficult even for strong methods.

Which fusion strategy should a multimodal architecture use?

There are three broad options, and production systems often mix them. The choice is a trade between detection power, explainability and how independently each part can be updated.

How should teams detect attacks spanning voice, video and documents?

Treat the case, not the file, as the unit of analysis. An attacker opening a fraudulent account may submit a forged ID document, a face-swapped selfie video and later a cloned voice on a support call. Each artifact should be analysed by the specialist model for its modality, and all results should attach to one case ID so the fusion layer and the reviewer see them together.

Then add cross-artifact checks that no single analyzer performs: does the face in the selfie match the document photo, do the voice on the call and the voice in the onboarding video belong to the same speaker, does the timing of submissions match a scripted pattern seen in other cases? This is what layered detection means in practice, and it depends on a shared case record with timestamps and model versions.

Documents need their own analyzers for template, font and pixel tampering. Face and voice models do not inspect them, and some products, including NeuroVerify, do not analyse ID documents at all, so the document check may come from a separate provider whose results must still join the case.

What should happen when a modality is missing or the modalities disagree?

Write the rules down. A video without an audio track, a call with no video, or an analyzer that failed on one input are common. The fused result should mark the modality as absent, not treat it as clean, and the policy should know whether the remaining evidence is enough for an automatic decision.

Disagreement is information. A clean voice with a manipulated face, or a genuine-looking document with a synthetic selfie, often indicates a partial attack. Surface the disagreement to the policy engine and reviewer instead of averaging it away. Monitor how often each modality is missing, excluded or in conflict, because a change in those rates can reveal a new attack or a broken capture path.

How should evidence be aligned before it is fused?

Fusion is only as good as its alignment. Video frames, audio samples and document pages arrive at different rates and sometimes from different sessions. Before combining anything, map each finding to a shared timeline and case, so a suspicious audio segment at 00:42 is compared with the frames at 00:42 rather than with a frame-level average over the whole clip.

Record the alignment itself. Reviewers and auditors need to know which segment of which file each score came from, which model version produced it, and whether any input was trimmed or resampled. Without that record, a fused result cannot be reproduced or challenged.

• Normalise timestamps to one clock at intake and keep the original offsets.
• Store per-segment scores, not only per-file averages.
• Version the fusion rules separately from the modality models.

How Neural Defend approaches multimodal detection

Neural Defend's products share ARCAS 1B, which its September 2026 arXiv paper describes as a foundational multimodal deepfake detection model. The paper reports pooled accuracy from 97.65% on OpenFake to 99.99% on GenImage, evaluated without benchmark-specific parameter updates. On the public VendorBench-100 still-image benchmark, Neural Defend ranked first of 36 systems with an MCC of 0.876, against 0.646 for the next vendor; that benchmark does not measure audio or video.

NeuroDetect is the multimodal workspace: an analyst uploads authorised images, audio, video and documents into one case, sees views down to regions, frames and audio segments, and exports a forensic report with uncertainty and model context. NeuroVerify brings six signals into one API call for a face image, video or audio clip and returns a unified risk score with signal-level evidence, so a policy engine can see which signal moved. NeuroCall (Beta) covers voice calls.

Integration with existing security tooling goes through the API and SDKs: Python, TypeScript and Node.js, Go and Java on the backend, Android, iOS and Flutter on mobile, a REST API and an MCP server for AI agents. Results can then be forwarded to your case management, fraud engine or SIEM by your own integration.