Real-time speech translation used to be the exclusive turf of expensive server-side pipelines: dedicated GPUs, WebSocket-heavy backends, and cloud vendors billing per minute of audio. It doesn't have to be. My Speech Translator ships a live in-browser translator with sub-second end-to-end latency, visible per-stage instrumentation, and a scripted-demo fallback for browsers without the Web Speech API. This post walks through how it works today and where AI takes it next.
The Latency Problem
Speech translation is really three latencies chained together: recognition (audio → text), translation (text → text), and synthesis (text → audio). Naïvely, you wait for a speaker to finish, transcribe, translate, synthesise, play. That's a two-second minimum before anyone hears anything. Conversational feel collapses beyond 800ms — the listener feels the lag, the speaker starts over-explaining, the flow dies.
The trick is to overlap all three stages. Start translating partial transcripts. Start synthesising partial translations. Show the user each stage happening so the perceived latency is closer to zero than the true latency.
Streaming ASR With the Web Speech API
The Web Speech API's SpeechRecognition interface exposes an interim-results stream that fires onresult events as the recogniser refines its hypothesis. Every frame gives you a slightly better guess at what the user said.
const rec = new webkitSpeechRecognition();
rec.continuous = true;
rec.interimResults = true;
rec.lang = "en-GB";
rec.onresult = (e) => {
const last = e.results[e.results.length - 1];
onPartial(last[0].transcript, last.isFinal);
};The catch: interim results flicker. "I want to" becomes "I wanted" becomes "I want to buy" — every revision is a valid translation target if you pipe it downstream naïvely. Do that, and you burn API tokens on translations that never should have existed.
LocalAgreement-2: Killing the Flicker
LocalAgreement-2 is a stabilisation trick from the streaming translation literature. The idea: hold each partial hypothesis until the next hypothesis confirms the same prefix. Only the confirmed prefix flows downstream to the translator.
function localAgreement(prev: string, curr: string): string {
const p = prev.split(" ");
const c = curr.split(" ");
let i = 0;
while (i < p.length && i < c.length && p[i] === c[i]) i++;
return p.slice(0, i).join(" ");
}Two consecutive hypotheses that agree on "I want to" release those tokens to translation. The remainder waits. Result: translations are stable, cache-friendly, and never rewrite themselves mid-sentence.
Gemini Flash on the Edge
The translation stage runs on Gemini Flash via a Next.js edge route. Flash is fast (sub-200ms typical), cheap, and — critically — supports streaming completions. The route pipes the stable prefix as fast as it's produced and streams tokens back to the browser.
export const runtime = "edge";
export async function POST(req: Request) {
const { text, from, to } = await req.json();
const stream = await gemini.streamGenerate({
model: "gemini-2.0-flash",
prompt: `Translate from ${from} to ${to}: ${text}`,
});
return new Response(stream);
}Edge deployment matters here. Every hop of TCP RTT is another 30–80ms of latency. Vercel Edge Functions place the model call close to the user; the browser sees translation begin appearing within one round trip of the confirmed prefix.
Speaking It Back: The TTS Pipeline
The final stage — text-to-speech — is the last place latency hides. Native SpeechSynthesis is free and instant but sounds robotic in most languages. Cloud TTS (ElevenLabs, Google WaveNet) sounds fantastic but adds 300–500ms and burns budget.
The compromise: use native TTS for interim playback, upgrade to cloud TTS for final segments. The listener hears something immediately, and quality settles as the sentence completes.
Instrumenting Everything
The demo overlay shows per-stage latency: ASR partial time, translation time, TTS synthesis time, network RTT. Not because users want to see it, but because building fast systems requires that you can see them being fast. When latency spikes, you know within one utterance whether the model, the network, or the recogniser is the bottleneck.
Where This Goes Next: On-Device AI
The browser is quietly becoming a competent AI runtime. Three technologies collapse the current architecture within a year or two.
WebGPU + ONNX Runtime Web. Whisper Tiny quantised to INT8 runs at faster-than-realtime on a modern laptop's integrated GPU, entirely in the browser. No API keys, no network, no privacy trade-off. The next iteration of Speech Translator will offer a WebGPU-only mode that uses Whisper for ASR and a small Gemma or Phi model for translation, all local.
WebLLM and MLC. MLC-LLM already compiles 7B-parameter models to run in-browser via WebGPU with reasonable throughput. As models get smaller and hardware gets better, an end-to-end on-device pipeline (ASR → LLM → TTS) becomes a Chrome flag rather than a research paper.
Streaming multimodal models. Gemini 2.5's live API and OpenAI's Realtime API skip the transcription-translation-synthesis pipeline entirely. You send audio, you receive audio, in the same language pair. The intermediate text is optional. When these APIs land at Flash pricing tiers, entire classes of translation apps become one WebRTC connection.
What AI Changes About the UX
Real-time translation today translates the words. Tomorrow it preserves the tone, the pause structure, the intent. Multimodal models can hear a hesitant "I guess..." and produce a translation that reads as hesitantly in the target language. Voice cloning means the translation can be spoken in the user's own voice, not a generic TTS baseline.
Agentic translation. Longer term, an agent layer sits above the raw translation: detecting when the speaker is joking, when they're being formal, when they're quoting an idiom that doesn't cross the language boundary. The model rewrites the translation to preserve meaning, not just tokens. This is where translation stops being a linguistic exercise and becomes a communication one.
Simultaneous interpretation. Human interpreters don't wait for a sentence to complete — they translate ahead, using context to guess what's coming. LLMs are starting to do this credibly. Combined with LocalAgreement-2 style stabilisation, the perceived latency drops from ~700ms to something under 300ms — genuinely conversational.
The Takeaway
The Speech Translator started as a latency-engineering problem: how do you hide network round trips inside natural speech pauses? The answer today is a pipeline of streaming APIs stitched together with careful stabilisation. The answer in two years is a single multimodal model running on the user's GPU. Either way, the browser wins.
Try it live: [speech-reco-seven.vercel.app](https://speech-reco-seven.vercel.app/).