What Are Multimodal Deepfakes?

What Are Multimodal Deepfakes?

A multimodal deepfake is a deception in which more than one type of media has been generated or altered to support the same false claim. The narrow case is a single video where both the face and the voice are synthetic. The wider case is a package: a cloned-voice phone call, a generated headshot on a messaging profile, a forged employee ID and an AI-written email, all pointing to the same invented or impersonated person.

The term matters because most detection tools, and most people, check one thing at a time. A finance clerk who doubts an email may be reassured by a voice that sounds right. A recruiter who doubts a CV may be reassured by a face on camera. Each channel lowers the suspicion raised by the others.

How Are Multimodal Deepfakes Built?

Attackers chain separate generators. A face-swap or reenactment model maps the target's face onto an actor's head movements. A voice clone, trained on a few minutes of public speech, converts the actor's voice or reads a script. A lip-sync model then adjusts the mouth so it matches the new audio. Large language models write the surrounding messages and translate them into fluent English or any other language.

Real-time versions run the face and voice models live over a virtual camera and a virtual microphone, so the impostor can answer questions on a video call. Pre-recorded versions allow more processing time and usually look cleaner. Researchers model the problem the same way: the FakeAVCeleb dataset, presented in the NeurIPS 2021 Datasets and Benchmarks track, pairs deepfake celebrity videos with lip-synced synthetic audio so detectors can be trained and tested on joint audio-video fakes.

The combinations form a grid. Real video can carry fake audio, fake video can carry real audio, or both can be fake. A detector that only checks the face will pass a real face with a cloned voice, and a voice detector will pass a face swap that kept the original audio.

Where Have Multimodal Deepfakes Caused Real Losses?

The Arup case is the clearest example. In January 2024 a Hong Kong finance employee who had doubted a phishing email joined a video call with people who looked and sounded like the company's chief financial officer and other colleagues. Hong Kong police said all of them were deepfake recreations. The employee then made 15 transfers totalling HK$200 million, about US$25.6 million, and Arup confirmed to CNN in May 2024 that fake voices and images were used.

Hiring fraud showed the same pattern earlier. On 28 June 2022 the FBI's Internet Crime Complaint Center warned of applicants using deepfakes and stolen personal information to apply for remote IT and software roles, and noted that interviewees' lip movements and actions did not fully match the audio, with coughs and sneezes out of step with the picture. In December 2024 the IC3 added real-time video chats impersonating executives and police, private videos sent to prove an online contact is real, and cloned voices of account holders used to reach bank accounts.

How Do You Detect a Multimodal Deepfake?

Detection works on two levels. First, test each channel with a method built for it: face manipulation and generator analysis for frames, spectral and prosody analysis for speech, document forensics for IDs, and reverse image search for profile photos. Second, test the joins. Lips should form the shapes the phonemes require, voice timbre should fit the face's apparent age and the room's acoustics, and light on the face should match the scene.

Cross-modal mismatches are valuable because the forger has to get every pairing right, while the defender needs only one solid inconsistency. The FBI's 2022 note about coughs that did not line up with the video is a cross-modal clue that a single-channel check would miss.

Scores from separate detectors cannot simply be added up into one probability. Treat each as evidence, record which channel and which segment it refers to, and let a fusion step or a reviewer weigh them. Then step outside the media entirely: call back on a number you already hold, confirm through a second approver, or hold the payment.

What Are the Limits of Detecting Multimodal Deepfakes?

Live calls are hard. Video conferencing compresses frames, drops packets and lowers resolution, which removes fine traces and creates sync glitches in genuine calls. A short call gives a detector few seconds of usable speech.

Media from the wild is harder than lab data. Deepfake-Eval-2024 (arXiv 2503.02857), collected from 88 websites in 52 languages, found open-source detectors lost about 50% of their AUC on video and 48% on audio compared with earlier benchmarks. A genuine channel can also be included on purpose: an attacker may use a real recording of an executive's voice from an earnings call and fake only the face.

No detector identifies the operator, establishes intent, or confirms that the transaction request is legitimate. Those answers come from process controls and investigation.

How Does Neural Defend Handle Multimodal Deepfakes?

NeuroDetect lets an investigator put every channel of an incident into one case: the recorded video, the extracted audio, the profile photo and the ID document. Each file is analysed with model and version context recorded, and explainability views point to the regions, frames and audio segments behind each finding, so the analyst can say which channel was synthetic and where. An LLM-assisted summary links its statements to that evidence, the analyst makes the final call, and the case exports as a forensic report with observations, uncertainty and provenance.

For automated decisions, NeuroVerify accepts a face image, video or audio in one API request and returns a unified risk score with signal-level evidence from six signals, including Face Manipulation Detection, AI Generative Model Analysis and Live Presence Verification. Voice and call deepfake detection is available in NeuroCall (Beta). The detection model underneath, ARCAS 1B, reports pooled accuracy between 97.65% and 99.99% across six public benchmarks in the ARCAS paper. Neural Defend deploys as a managed cloud API, in a private cloud or VPC, on premises or air-gapped.