I'm spending a few months in Asia, and for the last few weeks I've been building my own translator for the iPhone. The other person speaks, and I hear them in Polish, in my earbuds. It can run fully offline: speech recognition, translation and the Polish voice all happen on the phone, and no audio leaves it.
A few days ago I wrote down an observation I didn't like. In a taxi, in a busy city centre, sometimes even in a quiet room, Google Translate seemed to hear people better than my app did. The driver was speaking English. My app showed a warning: three empty recognitions in a row.
My first thought was: wrong model, get a better one. That thought was wrong. And the reason it was wrong is the most useful thing I've learned so far about putting AI into the physical world.
I thought this was about translation
When I started, I treated the project as one task: translate what the other person says. It turned out to be a chain. The microphone picks up the voice and everything around it. Something has to decide whether this sound is speech at all, and where a sentence ends. Then which language it is. Only then does a recognition model turn sound into text, and only then does translation start. After that come a queue, a voice in your ear, and everything a phone does to you: the screen locks, the device heats up, the battery drains, another app takes the audio.
Every link can lose information or add delay. And the language model, the part everyone talks about, sits somewhere in the middle.
What the logs actually said
I had no recording of the taxi ride. I don't record conversations. But the app keeps short technical logs on the phone, and they were enough to change the question.
Nineteen recognitions started. Every single one ended on the six-second safety limit. Not one ended because the app had found a pause. Of the eighteen that finished, thirteen came back empty. The microphone level was fine; this wasn't a phone in a pocket.
So the app wasn't deaf. In the noise it couldn't find where a sentence ended, so it cut speech at fixed intervals and handed the model fragments that started and stopped in the wrong places. I can't prove that every empty fragment contained the driver. It's a strong lead, not a verdict. But from the outside this looks exactly like a weak model, and the cause was one step earlier.
That's the first lesson, and I think it applies to every agent that will live next to people: a hearing error looks like a thinking error. If an agent misunderstood you, its reasoning may be fine. It may simply have reasoned over half a sentence.
A hearing error looks like a thinking error.
Why we don't see it
As a psychologist, I find this part the most interesting. None of these requirements were in my head when I started, because people do all of it without noticing.
In 1953 Colin Cherry described what we now call the cocktail party problem: in a room full of voices, we follow one speaker and filter out the rest. In 1970 Richard Warren showed that when part of a word is replaced by a cough, listeners still "hear" the missing sound and often can't tell where the cough was. The brain fills the gap from context. And we know, without thinking about it, that a conversation isn't over just because nobody has said anything for a while.
In robotics this is called Moravec's paradox: what's easy for humans is often the hardest part for machines, and the other way round. A model can summarise a contract. Knowing that the driver is about to speak again after ten minutes of radio is harder.
I see the same thing when I map business processes with people. The steps nobody mentions are the ones they do without thinking. They are also the steps that break first when you automate. The physical world has those steps too. We just never had to write them down.
A better component that made things worse
The fix I tried was a small voice activity detector, Silero VAD. It's a model of about 2.3 MB that runs on the phone and answers one narrow question: is this speech? The old method asked a different question: is this loud? In a taxi, a lot of things are loud.
I didn't want to judge it by one good sentence. So I built a repeatable test on the phone: the same recordings through the old method and the new one, four languages, normal speech, very quiet speech, white noise at several levels, a low hum, pauses, and sounds with no speech at all. Then a second test with a real microphone: my MacBook played the sentences out loud, and the iPhone listened from next to it.
The file tests looked good. With very quiet speech, the old method dropped every sample in all four languages. The new detector passed the speech through in all of them. That doesn't mean the text was right every time: in Thai and Filipino, recognition still made mistakes. On pure noise, the detector correctly sent nothing to the model, where the old method had produced a made-up word.
The real microphone told a different story. With the detector on, the Whisper model missed the noisy sentence completely, the same one the old method had caught. Its word error rate on that test went from 24% to 52%. The Parakeet model got the quiet sentence right for the first time, but cut the noisy one short; its overall result didn't move.
The same component helped in quiet and hurt in noise. That's not a result I want to hide. It's the most useful result of the week, because it shows where the next work is: the detector catches the start of a noisy sentence too late, so I need to test how much audio to keep before speech begins, and whether noise reduction before recognition helps.
The detector is still on by default in the travel version, with a switch back to the old behaviour. I didn't write "we solved noise", because we didn't.
Fifteen minutes of silence is still a conversation
While testing, I noticed two situations from my own days that no benchmark had.
The first: physiotherapy. Fifteen minutes in which nobody says anything, then the therapist says one important sentence. The second: a taxi. A long silence, the radio on, and at the end the driver turns around and asks something. In both cases the app should still be listening, catch that first sentence and translate it, without me reaching for the phone to press Start again.
Many apps treat a long silence as the end of the session, to save battery. For an agent that is supposed to stay with you, that's exactly wrong. My app now has a "listen until I stop it" setting, on by default, which removes the idle limit. A logic test of fifteen minutes of silence followed by speech passes. A real fifteen minutes through the microphone, with the screen locked, I haven't done yet.
There's a subtle point here too. Silence and a broken microphone look alike: no speech. But silence means audio is still arriving and nobody is talking. A broken audio route means no audio arrives at all. One is normal, the other is a failure, and the app must not confuse them.
And the radio. At first I assumed the app should ignore it. It shouldn't. The radio can be translated; that's fine. What matters is that when the driver starts talking, his sentence isn't stuck in a queue behind three minutes of news. A radio playing at the same moment the driver speaks is a different and harder problem: the detector knows there is speech, but it can't separate two voices.
Offline is not the same as better
I like that the app works without a network. It's private, and it works where there's no mobile data. But offline gives you independence, not better hearing. Google may have better noise handling, better segmentation or better models; I don't know how their system works, and my comparison was my own experience, not a controlled test. "It runs on the device" is a privacy property. It says nothing about whether it understood the driver.
What I'm taking into agent design
Szept is a translator, not an agent. It doesn't act in the world. But it's the first project where I'm building the layer every "always with you" agent will need: perception that works among real people, real noise and real pauses. A few rules I now follow:
- Test the chain, not the model. The same input through the old and the new version, with a real microphone in the test. One good sentence proves nothing.
- Write down the requirements people never say out loud. Long silence, interruptions, a third person talking, a radio, a locked screen. Ask about the last concrete situation, not about how it usually goes.
- Separate "nothing happened" from "I can't tell". Silence is not a missing signal. An empty transcript is not proof that nobody spoke.
- Keep the regression in the story. The case where the new part made things worse is the map for the next iteration.
- Judge the timing, not only the answer. A correct translation that arrives after three minutes of radio is a wrong translation.
I keep coming back to one thought. I forgive a person who says "sorry, I didn't catch that". I'm much less patient with someone who confidently answers something I never said. For an agent at your side, knowing what it missed may matter more than how well it reasons about what it heard.