Live speech-to-text is the one feature where I use WebSockets instead of Server-Sent Events, because audio flows up while text flows down. The OCR & Speech Workspace does live microphone transcription with a post-recording refinement pass, and the architecture below is what made it feel instant rather than laggy.
The data flow
- The browser captures microphone audio, resamples to the rate the model expects, and sends small chunks — a few hundred milliseconds each — over a WebSocket.
- The server buffers chunks into windows, runs the streaming model, and sends back partial transcripts as they stabilise.
- When the user stops, the server runs a second, slower pass over the whole recording and replaces the live text with a refined final transcript.
Partial versus final
Streaming models revise their guesses as more audio arrives. Show partial text in a lighter style and lock segments once they are final; users tolerate words changing for a second, not paragraphs rewriting themselves. Mark segment boundaries with silence detection so the refinement pass has clean units to work with.
Latency budget
- Chunk size sets the floor: 300 ms chunks mean nothing appears for at least 300 ms.
- Network round trip and model time add to it; keep the model warm and the connection persistent.
- Aim for under a second to the first partial; beyond two seconds, users start repeating themselves.
The refinement pass is not optional
Live transcripts are optimised for speed, not accuracy. A post-recording pass with a stronger model, punctuation restoration and speaker-aware formatting turns a stream of words into a document someone will actually read. In the workspace, that final transcript is what gets indexed for document-scoped chat.
Details that matter
Ask for microphone permission at the moment of need with an explanation, show a live level meter so the user knows audio is flowing, handle reconnects without losing the buffer, and let people edit the final text. Speech features live or die on these details, not on the model.
About Pranjul Rathour
Presenting KrishGyan
Pranjul Rathour — GenAI engineer, Kanpur
Speaking at a MeetKats event
Taking questions during a session
At VSICS, Kanpur
Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-06.
Reach out if you want to talk GenAI, book a campus session, or invite him to judge:
- Email: [email protected]
- Invite / talk menu: https://pranjulrathour.scult.in/invite
- Portfolio & blog: https://pranjulrathour.scult.in
- LinkedIn: https://www.linkedin.com/in/pranjul-rathour/
- X: https://x.com/PranjulRathourx
- Instagram: https://www.instagram.com/pranjulrathour.in/
- Bluesky: https://bsky.app/profile/pranjulrathour.bsky.social
- GitHub: https://github.com/Pranjulrathour
Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: [email protected] · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to