02EvaluationFrom the lab to the real world
From the lab to WhatsApp: how I made a cloned-voice detector reliable in the real world
In the lab, almost every detector works well. The hard part comes when the voice goes through a phone call or a WhatsApp voice note. How I took mine out of the lab.
The case
On a Friday afternoon, a company assistant receives a WhatsApp voice note from an unknown number. It's the CEO's voice: he asks for an urgent bank transfer, to be made right away. But the voice is fake: an AI generated it.
Before reaching the phone, that voice note has travelled. The app compresses it, the network trims it, sometimes someone plays it on speaker and records it again. Every step erases details, and those are exactly the details a detector uses to recognise a synthetic voice.
Original
The file as it comes out of the generator
WhatsApp voice note
The same file, passed through WhatsApp
Phone call
The same file, passed through a phone call
The problem: the lab is not the world
In 2025 a public study tested the best open-source detectors on content actually taken from social media. On audio, they lost about half of the performance claimed in the lab. Many ended up close to guessing.
The reason is almost always the same: the detector is measured on audio too similar to what it was trained on. The grade is high, but it was earned on homework it had already seen.
A report card with three grades
That's why I never measure a detector with a single grade. I give it three, and each one answers a different question:
- New voices and recordings. Does it work on people, languages and microphones it has never heard?
- New AI tools. Does it recognise voices made with generators it doesn't know? This is what matters when a new tool comes out.
- Real world. Does it hold up on audio that is compressed, forwarded and badly recorded, like what actually arrives?
I never average the three grades. A detector can score 90 on the first and 40 on the third: the average, 65, tells you nothing useful. Whoever uses the detector needs to know where it is strong and where it isn't.
The threshold that doesn't travel
A detector doesn't answer "real" or "fake". It gives a score, and above a certain threshold the audio is flagged as fake. Where you set the threshold decides how many false alarms you get.
It's like a smoke alarm. Calibrated for an office, it works perfectly. Installed in a kitchen, it goes off every time someone starts cooking.
It really happened to me. I had calibrated my first threshold on my own test data. On a public collection of almost 2,000 audio clips taken from social media, it flagged 19 real voices out of 100 as fake. Almost one real person in five accused of being an AI.
Real voices recorded in a studio
5 out of 100 flagged as fake
Real voices from social media
19 out of 100 flagged as fake
The idea: teach the model the journey
I did two things.
First: the model learns what a voice sounds like after the journey. During training it hears voices, real and fake, also as they would arrive in reality: in different rooms, with background noise, through phone calls and messaging apps. So it learns that a compressed voice is not necessarily a fake one.
Second: the threshold is calibrated on the type of audio being judged. And in the report I always state which type of audio it was calibrated on, so the reader knows how far to trust it.
The proof
And the three-grade report card today:
The lesson
A detector is only as good as it is outside the lab. That's why I test mine on the type of audio it will have to judge, before it reaches someone who has to make a decision: a company about to make a bank transfer, a lawyer, a judge.
In short
- I measure a detector where it will be used, not only in the lab.
- I keep the three grades separate: new voices, new tools, real world.
- The threshold is calibrated on the type of audio being judged.