What Neural Vocoders Contribute to Synthetic Speech Realism

What neural vocoders do in synthetic speech

A neural vocoder is the last stage of most speech synthesis and voice cloning systems. Earlier stages decide what should be said and how it should sound, usually as a mel spectrogram: a compact picture of which frequencies carry energy in each few-millisecond frame. That picture cannot be played. The vocoder fills in everything it leaves out, chiefly phase and fine harmonic detail, and outputs tens of thousands of audio samples per second.

Realism lives in that missing detail. Classical vocoders reconstructed waveforms with signal-processing rules, and listeners heard the result as buzzy or metallic. Neural vocoders learn the mapping from recorded speech, so breath, voicing and the texture of consonants come back much closer to a real recording. When people say a cloned voice sounds human, a large share of the credit belongs to the vocoder.

WaveNet: realism at a heavy computational cost

DeepMind's WaveNet paper, submitted on 12 September 2016, generated raw audio one sample at a time, with each sample conditioned on all previous ones. In listening tests for US English and Mandarin, people rated it more natural than the best parametric and concatenative systems of the day.

The cost was speed. Producing tens of thousands of samples per second sequentially was far too slow for a product. DeepMind's October 2017 production version ran about 1,000 times faster than the original, generating one second of 24 kHz speech in 50 milliseconds, and shipped in Google Assistant voices.

Tacotron 2 (December 2017) then paired a sequence-to-sequence spectrogram predictor with a modified WaveNet used purely as a vocoder. It scored a mean opinion score of 4.53 against 4.58 for professionally recorded speech, which settled the two-stage design that most voice cloning tools still follow.

GAN vocoders made high-quality speech cheap

Generative adversarial network (GAN) vocoders removed the sequential bottleneck by generating all samples in parallel. A generator upsamples the spectrogram to a waveform while discriminators judge whether the result sounds like real audio, which pushes the generator towards natural texture.

HiFi-GAN, published in October 2020 and accepted at NeurIPS 2020, became the reference design. Its authors reported generating 22.05 kHz audio about 167.9 times faster than real time on a single V100 GPU, and a small version ran about 13.4 times faster than real time on a CPU. Their key observation was that modelling the periodic patterns of speech is crucial to quality, so HiFi-GAN uses discriminators that look at the waveform at several periods.

Real-time synthesis on ordinary hardware is what made live voice conversion and interactive voice bots feasible. Once a vocoder can keep pace with a phone conversation, a cloned voice can respond turn by turn instead of playing back prerecorded lines.

Neural audio codecs: where the vocoder role is heading

Since 2023 a second design has spread alongside the spectrogram-plus-vocoder pipeline. Neural audio codecs compress speech into sequences of discrete tokens, and a codec decoder plays the vocoder's part by turning tokens back into a waveform. Microsoft's VALL-E, published in January 2023, predicts these codec tokens with a language model and clones a voice from a 3-second enrolled recording.

For defenders, the change matters because the decoder leaves a different fingerprint from a GAN vocoder. A detector that learned HiFi-GAN artefacts well may score codec-generated speech less confidently. Evaluation sets therefore need samples from both families, and from speech-to-speech converters that reuse the same decoders.

Do vocoders leave traces that detectors can find?

They do, which is why vocoders sit at the centre of audio deepfake detection. Each architecture reconstructs phase and high-frequency detail in its own way, and the errors are systematic: unnatural phase continuity, periodic artefacts from upsampling layers, and energy patterns in high frequency bands that real microphones and vocal tracts do not produce. Listeners rarely notice these; trained classifiers often do.

The weakness is generalisation. A detector trained mostly on one vocoder family can learn that family's fingerprint and then miss a new one. The ASVspoof challenge series exists to measure this; its 2024 edition, ASVspoof 5, used attacks generated with many systems and added adversarial attacks for the first time. Phone codecs complicate things further, because band-limiting and compression erase part of the high-frequency evidence.

What vocoder progress means for call and media security

Vocoder quality is no longer a meaningful barrier for attackers. Open-source GAN vocoders run in real time on laptops, so the realism of a cloned voice is limited mainly by the quality of the reference audio and the attacker's script. Defences have to assume the audio will sound right.

In practice that means three things. Treat voice alone as insufficient proof of identity for payments, password resets or account changes. Use detection that is retrained as new generators appear and tested on narrowband call audio, not only studio clips. And keep the decision with policy: a detector score should trigger a callback or step-up check rather than an automatic accusation.

Neural Defend's NeuroCall, currently in Beta, applies this to live and recorded calls, returning a risk signal on synthetic speech for your call-handling rules. Its detection work builds on ARCAS 1B, Neural Defend's own deepfake detection model. For disputed recordings, NeuroDetect lets an analyst inspect audio-segment level evidence and export a forensic report with model and version context.