Parameter- and Bandwidth-Efficient Edge–cloud Many-to-Many Speech-to-Text Translation

[Submitted on 27 May 2026 (v1), last revised 14 Aug 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Multimodal large language models (MLLMs) have demonstrated significant potential for speech-to-text translation (S2TT). However, existing deployment paradigms face critical challenges: pure on-device models suffer from resource constraints, while centralized cloud systems incur bandwidth bottlenecks and privacy risks by transmitting raw voice data. In this paper, we propose Edge–cloud Speech Recognition and Translation (ESRT), a parameter-efficient, bandwidth-efficient, and privacy-aware collaborative Edge–cloud MLLM framework. First, we introduce a multi-task weighted curriculum learning strategy to mitigate catastrophic forgetting, improve multilingual balance, and train parameter-efficient ESRT-1B, ESRT-4B, and ESRT-12B models. Second, we enable bandwidth-efficient Edge–cloud inference by retaining a lightweight speech encoder and adapter on the device and transmitting only a compressed tensor to the cloud. Extensive experiments on FLEURS demonstrate that ESRT models achieve state-of-the-art S2TT performance across 45 languages ($45 \times 44$ directions). Relative to raw audio, ESRT and ESRT-Lite reduce the transmitted tensor size by $5.1\times$ and $10.2\times$, respectively, while keeping raw speech on-device and avoiding its direct exposure to the cloud. The code and models are released to facilitate reproducible, privacy-aware S2TT research.

Submission history

From: Yexing Du [view email]
[v1] Wed, 27 May 2026 15:47:33 UTC (8,863 KB)
[v2] Fri, 14 Aug 2026 02:21:24 UTC (11,658 KB)