How Should Synthetic Media Realism Be Measured?

How should synthetic media realism be measured?

Synthetic media realism should be measured as a profile across named dimensions, under stated viewing and delivery conditions, against the decision the media is meant to influence. Image detail, identity fidelity, lip sync, head and body motion, temporal stability, lighting, physics, speech naturalness and fit with the delivery channel all improve on different timelines. A clip can be excellent on three of them and fail a fourth, and the failing one may be the only one that matters for a given decision.

Two further distinctions keep measurement honest. Realism to a human viewer is different from detectability by a machine. And realism in a lab, at full resolution on a large screen, is different from plausibility in a compressed voice note or a small phone video. A useful measurement says which of these it is reporting.

Why do single realism scores mislead?

Single scores compress many properties into one number and hide which one failed. Fréchet Inception Distance, introduced in June 2017 by Martin Heusel and colleagues, compares the statistics of generated and real images in the feature space of an image classifier; lower is better. It became the standard image metric for GANs, but it says nothing about whether a specific face matches a specific person. Fréchet Video Distance, proposed in December 2018, extended the idea to video and was shown in a large human study to correlate with human judgements, yet it still returns one value.

VBench, released in November 2023, took the opposite approach and split video generation quality into 16 dimensions, including subject identity inconsistency, motion smoothness, temporal flickering and spatial relationships, each with tailored prompts and checks aligned with human preference. That design reflects the real situation: generators advance unevenly, and a single number can rise while a property that matters to you gets worse.

What human perception studies actually measure

Perception studies measure whether people can tell real from synthetic under specific conditions. The best-known result comes from a 2022 PNAS study by Sophie Nightingale and Hany Farid using StyleGAN2 faces. With 315 participants, accuracy was 48.2%, near chance. A trained group of 219 reached 59.0%. A third group of 223 rated synthetic faces as more trustworthy than real ones, 4.82 against 4.48 on a seven-point scale.

Those numbers describe still faces viewed in a study, not every modality or setting. When reporting human realism, state the task, the number of raters, the viewing time and display, whether raters knew fakes were present, and how samples were selected. Change any of these and the result changes.

Why machine detectability is a separate axis from realism

A fake that fools people can still be easy for a detector, and a crude-looking fake can sometimes slip past one. Detectors read statistical traces that people cannot see, while people notice contextual contradictions a detector never receives. So detectability needs its own measurement, with metrics suited to imbalanced data. The VendorBench-100 authors rank 36 systems by Matthews correlation coefficient on 100 adversarial still images and warn that accuracy and F1 mislead on such sets; they also show that a high ROC-AUC can coexist with poor performance at the default threshold, as with TruthScan's 0.923 ROC-AUC and 0.322 MCC.

NIST AI 100-4, published in November 2024, treats detection, provenance, watermarking and human review as complementary approaches. That is the right frame: a realism profile tells you how convincing a piece of media is; a detectability measurement tells you whether your controls will catch it.

How the channel and the decision set the bar

The realism needed to cause harm depends on where the media arrives and what it asks for. A short, compressed voice message claiming an emergency may need only a passable voice and urgency; the FBI's December 2024 announcement describes cloned voices of relatives asking for money. A video submitted as evidence in a dispute needs far stronger continuity and a credible source history. Measuring a fake at studio quality overstates the bar for the first case and understates it for the second. Test media after the same compression, resolution and display it will have in the real channel.

What a useful realism assessment should contain

A report that others can rely on records enough to reproduce the judgement and to see where it stops.

• The generating system and version, and the input or reference material used.
• Modality, duration, resolution and the channel processing applied before assessment.
• How samples were selected, including how many attempts were discarded.
• Each dimension measured, the method used, and the result with its uncertainty.
• For human ratings, the task, raters, viewing conditions and whether fakes were expected.
• For detectors, the model version, threshold and the data it was tested on.
• The decision the evidence supports, stated with its limits.

How Neural Defend measures detectability on your media

Neural Defend's position is that realism does not decide whether a fake gets caught; measured detectability on your own channel does. NeuroVerify returns one risk score with signal-level evidence from six signals, including Face Manipulation Detection, AI Generative Model Analysis and Live Presence Verification, so you can see which property triggered a flag rather than a bare verdict. NeuroDetect's forensic reports record observations, uncertainty, provenance and model context for each case, the kind of per-dimension record this page recommends.

On the public VendorBench-100 benchmark, Neural Defend ranked first of 36 systems with MCC 0.876, precision 0.963, recall 0.987 and specificity 0.857. One VendorBench author also co-authored Neural Defend's ARCAS 1B paper; the corpus and results are public so anyone can rerun them. The benchmark measures still-image detection only, so pair it with tests on your own video and audio.