What Is Voice Cloning? How It Works, Risks and Protection

What is voice cloning?

Voice cloning is the use of AI to create a synthetic copy of a specific person's voice, so a computer can say new words that sound as if that person spoke them. The clone captures timbre, pitch range, accent and often speaking style. It can read typed text aloud, or it can transform someone else's live speech into the target voice.

What changed in the 2020s is the amount of audio needed. Research systems now work from seconds of speech: Microsoft's VALL-E paper (January 2023) describes cloning an unseen speaker from a 3-second recording, and Starling Bank warned UK customers in September 2024 that criminals can clone a voice from three seconds of audio taken from online videos. A voicemail greeting, a podcast clip or a social media video can be enough.

How voice cloning works

There are two main routes. Text-to-speech cloning trains a model on thousands of voices, then conditions it on a short sample of the target. A speaker encoder summarises the sample as an embedding, a synthesiser predicts how the target would say the input text, and a vocoder produces the audio. Google's SV2TTS design from 2018 established this pattern, and codec language models such as VALL-E pushed reference requirements down to seconds.

Voice conversion keeps a human in the loop. The attacker speaks, and a converter maps their voice onto the target's in near real time while keeping the words, timing and emotion. This makes interactive calls possible, because the attacker can answer questions naturally instead of playing scripted lines.

Both routes rely on neural vocoders or audio codecs to produce the final waveform, and both leave statistical traces in the result. Those traces are what automated detectors look for.

Legitimate uses and harmful uses of voice cloning

Voice cloning has real benefits. People who are losing their voice to illness can bank it for later use, creators can dub content into other languages in their own voice, and studios can fix a line without a re-recording session. These uses depend on the speaker's consent and control over where the clone is used.

The harms come from cloning without consent. The FTC warned in March 2023 that scammers use cloned voices of relatives in fake family emergencies. The FBI's December 2024 announcement says criminals use cloned audio to impersonate family members, public figures and bank customers, and its May 2025 alert described AI voice messages impersonating senior US officials. In a Starling Bank survey published in September 2024, 28% of UK adults said they had been targeted by an AI voice cloning scam in the past year, and 46% had never heard of the scam.

Is voice cloning legal?

Cloning your own voice, or a voice you have permission to use, is legal in most places. Using a clone to deceive or defraud is not. In the US, the FCC confirmed in a declaratory ruling released on 8 February 2024 that the TCPA's restrictions on artificial or prerecorded voices cover AI-generated voices, so robocalls using them need prior express consent. Tennessee's ELVIS Act, passed in 2024, added voice to the likeness rights the state protects.

In the EU, Article 50 of the AI Act applies from 2 August 2026. It requires providers of generative AI to mark synthetic audio in a machine-readable way, and requires deployers to disclose audio deepfakes to the people who hear them. None of these rules stops a criminal from making a clone, so detection and verification remain necessary.

How to protect yourself and your organisation

For individuals, the most effective protections are simple. Agree a code word with family, as the FBI recommends. When a call asks for money or secrecy, hang up and call back on a number you already have. Consider limiting public recordings of your voice, and make social accounts private where you can.

For organisations, the key rule is that a familiar voice never authorises a sensitive action by itself. Payments, bank detail changes, password resets and new device enrolments should require confirmation through a separate, pre-registered channel. NIST's SP 800-63B-4, finalised in July 2025, goes further and says voice biometric comparison shall not be used in its authentication guidelines.

The FTC's 2023 to 2024 Voice Cloning Challenge pointed to the technical side of defence. Its winning ideas included algorithms that tell human from synthetic voices, real-time scoring of incoming calls in two-second chunks, and inaudible watermarks that make voices harder to clone.

How cloned voices are detected

Detection models learn the difference between recorded human speech and synthetic speech from large sets of both, looking at phase, fine spectral detail and the fingerprints of specific vocoders and codecs. They are most accurate on clean audio and on generator families they have seen, and less accurate on narrowband phone calls and brand-new tools.

Neural Defend's NeuroCall, currently in Beta, applies voice deepfake detection to call audio and returns a risk signal your call-handling policy can act on. NeuroVerify accepts audio in the same API request as face images and video for onboarding and recovery checks, and NeuroDetect gives investigators audio-segment level evidence and a forensic report for recordings under review.