A compact speech recognition model built for edge devices — designed to run efficiently without heavy computing power — turns out to have a fundamental flaw when it encounters Bengali. Researchers Sanjid Hasan and Md. Abdur Rahman have pinpointed exactly why Bengali speech recognition breaks down in these lightweight systems, and their fix is surprisingly surgical: swap out the tokenizer vocabulary rather than retrain the entire model from scratch.
Summary
Key takeaways
- English-centric byte-level tokenizers fragment Bengali words into long, unstable sequences, triggering autoregressive collapse during inference.
- Vocabulary transplantation replaces the decoder vocabulary with BanglaBERT WordPiece and resizes the token embedding matrix, requiring no costly pre-training.
- Token fertility dropped from 9.16 to 1.30 after the transplant, and autoregressive sequence length fell by 85.8%.
- On the 882-hour Lipi-Ghor dataset, the modified model hit a 21.54% Word Error Rate and a Real-Time Factor of 0.0053.
- The research was accepted as a poster at the MusIML Workshop, ICML 2026.
Challenges in Lightweight Bengali Speech Recognition
Edge-deployable automatic speech recognition (ASR) has made remarkable strides, but those gains have been largely concentrated in languages that share Latin script and morphological simplicity with English. Bengali sits at the other end of that spectrum — it is morphologically rich, uses its own script, and proves resistant to models optimized almost entirely on English-language data.
Impact of Morphologically Rich Non-Latin Scripts
Moonshine, a highly optimized lightweight ASR architecture, is built to run on constrained hardware. That efficiency comes with trade-offs. When applied to Bengali, the model’s architecture — trained and tuned around English — struggles to represent the language’s morphological complexity. Words that carry significant meaning through inflection and script-specific characters simply do not map cleanly onto a system designed for a structurally different language.
The result is not just reduced accuracy. It is a fundamental decoding failure.
Problems with English-Centric Byte-Level Tokenizers
The root cause, according to Hasan and Rahman, lies in the tokenizer. Moonshine uses an English-centric byte-level tokenizer — a design that works well for English but treats Bengali characters as raw bytes rather than meaningful linguistic units. Bengali words get fragmented into long chains of high-fertility tokens, meaning each word explodes into far more individual tokens than any model should reasonably need to process.
This fragmentation is not merely inefficient. It triggers what the researchers describe as catastrophic autoregressive collapse: during inference, the model loses coherent decoding and produces unusable output. The problem is structural, not incidental, which is precisely why simply fine-tuning the existing model on Bengali data cannot fully resolve it.
Vocabulary Transplantation Method to Mitigate Autoregressive Collapse
Rather than rebuilding the model or running expensive pre-training on new data, the researchers propose an intervention at the tokenizer level — a method they call vocabulary transplantation.
Replacing Decoder Vocabulary with BanglaBERT WordPiece
The transplantation works by replacing Moonshine’s original decoder vocabulary with the BanglaBERT WordPiece vocabulary — a native-script vocabulary designed specifically for Bengali. BanglaBERT’s WordPiece tokenization understands Bengali morphology, grouping characters and subwords into units that actually reflect how the language is structured, rather than how bytes happen to be arranged.
This is a targeted surgical replacement, not a full architectural overhaul. That distinction matters enormously for practical deployment: the approach avoids the resource-intensive pre-training that would normally be required when adapting a model to a new language family.
Adjusting the Token Embedding Matrix
Swapping vocabularies alone is not sufficient. The token embedding matrix — the internal lookup table that maps tokens to numerical representations the model can process — must be resized correspondingly. Hasan and Rahman include this step as part of the transplantation pipeline, ensuring that the model’s internal representations align with the new vocabulary rather than producing mismatched or undefined embeddings.
Together, these two changes — vocabulary replacement and embedding matrix resizing — constitute a complete, self-contained adaptation pipeline that requires no additional pre-training data.
Performance Improvements and Experimental Results
Reduction in Token Fertility and Sequence Length
The numbers tell the story clearly. Before transplantation, token fertility sat at 9.16 — meaning each Bengali word, on average, was being broken into more than nine tokens. After transplantation with BanglaBERT WordPiece, that figure dropped to 1.30. Each Bengali word now maps to just over one token on average, which is close to ideal for any tokenization scheme.
That reduction cascades directly into decoding stability. Autoregressive sequence length — the number of steps the model must take to decode an utterance — fell by 85.8%. With shorter sequences, the conditions that previously triggered catastrophic collapse are eliminated entirely.
Model Evaluation on Lipi-Ghor Dataset
Testing was conducted on the Lipi-Ghor dataset, an 882-hour Bengali speech corpus. The modified Moonshine architecture achieved a Word Error Rate of 21.54% — a competitive result for a lightweight, edge-optimized model evaluated on a large-scale dataset in a morphologically complex language.
WER measures the proportion of words that a model transcribes incorrectly, so lower is better. At 21.54%, the system is far from perfect, but it is functional and meaningful for real-world applications — a significant contrast to the outright decoding failures the base model produced before the vocabulary transplant.
Processing Efficiency Metrics
Speed matters just as much as accuracy for edge deployment. The modified model recorded a Real-Time Factor of 0.0053, meaning it processes audio dramatically faster than real time. An RTF below 1.0 indicates the system transcribes speech faster than it is being spoken; at 0.0053, the model is processing audio roughly 189 times faster than real time — a clear indicator that the vocabulary transplantation did not introduce significant computational overhead.
This combination of competitive accuracy and extreme speed efficiency is what makes the approach particularly relevant for edge applications — devices with limited processing power that still need to handle speech recognition reliably and quickly.
Why This Matters Beyond Bengali
The broader significance of this work extends past a single language. The researchers frame their pipeline as a scalable and reproducible blueprint for cross-script adaptation of compact ASR models. If the same tokenizer transplantation approach can be applied to other morphologically rich, non-Latin languages — think Arabic, Tamil, or Amharic — it opens a pathway for deploying efficient speech recognition in linguistic communities that have historically been underserved by English-centric AI infrastructure.
The research was accepted as a poster at the MusIML Workshop at ICML 2026, placing it within one of the most prominent machine learning research venues globally. That acceptance signals peer recognition of the method’s significance, even if the full implications for other languages and deployment environments remain to be tested. The central insight — that vocabulary is the failure point, not the model architecture itself — reframes how developers might approach ASR adaptation for low-resource, non-Latin languages going forward.
FAQ
Why do lightweight ASR models struggle with Bengali?
Bengali is morphologically rich and uses a non-Latin script. English-centric byte-level tokenizers fragment Bengali words into long chains of tokens rather than meaningful linguistic units, causing autoregressive collapse during inference and making coherent transcription impossible.
What is vocabulary transplantation in this study?
Vocabulary transplantation is a method that replaces the decoder’s original English-centric vocabulary with the BanglaBERT WordPiece vocabulary, which is designed for Bengali. The token embedding matrix is resized correspondingly, adapting the model to Bengali without requiring expensive pre-training from scratch.
How did the vocabulary transplantation affect token fertility?
Token fertility dropped from 9.16 to 1.30 after the transplant. This means Bengali words are now represented by just over one token on average instead of more than nine, dramatically reducing the sequence length the model must decode and eliminating decoding instability.
What performance gains were observed on the Lipi-Ghor dataset?
On the 882-hour Lipi-Ghor dataset, the modified model achieved a 21.54% Word Error Rate and a Real-Time Factor of 0.0053, demonstrating both competitive transcription accuracy and highly efficient processing speed suitable for edge deployment.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
