Translated captions in a Chrome extension: tab audio in MV3, and one stream instead of two services

I build a Chrome extension on my own that shows translated captions over a Meet, Zoom or Teams call running in a browser tab. No bot joins the call; the audio comes from the tab itself. Below are the three things that took the most time. Figures are as of September 17, 2026.

Tab audio in Manifest V3

In MV3 the extension background is a service worker, and it has neither Web Audio nor getUserMedia. So the service worker gets only a stream id and creates an offscreen document (reasons: [‘USER_MEDIA’]), an invisible page where those APIs exist:

const streamId = await chrome.tabCapture.getMediaStreamId({ targetTabId: tabId });
// inside the offscreen document:
const stream = await navigator.mediaDevices.getUserMedia({
  audio: { mandatory: { chromeMediaSource: 'tab', chromeMediaSourceId: streamId } },
});

Enter fullscreen mode Exit fullscreen mode

First trap: once the tab is captured, its sound stops reaching the speakers, and the user can no longer hear the call. The fix is to route the stream back to the output:

const ctx = new AudioContext();
ctx.createMediaStreamSource(stream).connect(ctx.destination);

Enter fullscreen mode Exit fullscreen mode

An AudioWorklet then mixes the channels down to mono and emits Int16 PCM in blocks of 4096 samples, about 85 ms at 48 kHz.

Second trap: “Cannot capture a tab with an active stream”. It shows up when the previous capture of that tab was not released, and neither a page reload nor chrome.runtime.reload() clears it. The stream lives in the offscreen document, so that document has to be closed. Even that is not enough: Chrome releases the stream asynchronously, so poll the state instead of guessing with a timeout:

// poll every 50 ms, for at most 500 ms, after closing the offscreen document
const tabs = await chrome.tabCapture.getCapturedTabs().catch(() => []);
const busy = tabs.some((t) => t.tabId === tabId && t.status === 'active');

Enter fullscreen mode Exit fullscreen mode

Third, a product one: getMediaStreamId needs the extension to have been invoked on that tab (activeTab), and the grant only lasts until navigation. After a page reload the honest message is “click the icon”, not “error”. And only tab audio can be captured: desktop Zoom and Teams apps, like phones, are out of reach.

Two services versus one stream

The first version recognised speech with a Deepgram stream and translated with DeepL. To keep the translation from appearing in one jump at the end of a sentence, interim results were translated too. DeepL bills by source length, and an unfinished line keeps growing and gets sent again in full. How expensive that is cannot be read from the code, it depends on how recognition splits the phrase, so I measured it on the wire: a test track went into a real Deepgram socket, and interim results ran through the extension engine with the translation call stubbed. Measured on August 29: 2.53 times more characters went to translation than when translating final phrases only, and an hour of call cost about $2.55.

The way out was a provider that recognises and translates in one stream. I moved to Soniox, model stt-rt-v5. The server mints a temporary key: the TTL limits how long the key can open streams, not how long an open stream lives, so a call of any length fits a TTL of a few minutes. The session config goes as the first message, before the first byte of audio:

{
  api_key, model: 'stt-rt-v5',
  audio_format: 'pcm_s16le', sample_rate: 48000, num_channels: 1,
  language_hints: ['en'], enable_language_identification: true,
  enable_endpoint_detection: true,
  translation: { type: 'one_way', target_language: 'es' },
}

Enter fullscreen mode Exit fullscreen mode

What the docs did not say:

1. The server announces the end of a line with an token, but translation lags a few tokens behind speech. Closing the line right at glues the translation tail to the next phrase, so I hold a 700 ms window.
2. Language codes come without regions: pt-BR and pt-PT both become pt, zh-Hans and zh-Hant both become zh.
3. Errors 400, 401, 402 and 403 are not fixed by reconnecting; retries only give almost twenty seconds of empty screen. I treat them as fatal at once and fall back to a slower pipeline of 3-second chunks on another provider’s key.

What an hour costs, by the invoice

From /v1/usage/summary, August 30 to September 17: stt-rt-v5, $31.23 for 202.6 hours of audio, which is $0.154 per hour. Against $2.55 with two services, that is 17 times less.

The price is paid in latency, not money. On the test stand in August a final phrase arrived after a bit over 2 s with Deepgram; on September 8 with Soniox it took 4.3 and 7.6 s, because the server now decides where the phrase ends.

Testing without live calls

Everything interesting happens in Chrome and on the real platform, so unit tests catch little. A Linux box runs a stand: three Chrome profiles join one call and “speak” with their own voices (an interview script of 118 lines, synthesised in advance), and a fourth profile with the extension writes captions. A script compares them with the script: how many lines were recognised, under which speaker name, how many milliseconds until the final.

That is how the latency difference between providers showed up, and a billing bug: time was counted from the socket opening, not from the first phrase, so a silent tab burned minutes. Now the counter waits for the first final: 43.7 s of open socket on a silent tab, zero seconds billed.

Still unsolved

Final latency is higher than I would like for a conversation. And I cannot yet tell “the tab is silent” from “the provider is quiet”: the PCM counter grows during silence too.

I would be glad to hear critique of the MV3 capture part, especially if someone found something more reliable than polling getCapturedTabs

I wrote this with help from an AI assistant for translation and editing. The code, measurements and figures come from my own project.