⚡ electrical engineering

MOSS Transcribe Diarize Technical Report

MOSS Transcribe Diarize is a unified multimodal large language model that achieves state-of-the-art speaker-attributed, time-stamped transcription by leveraging a 128k context window and end-to-end training on extensive real-world data to overcome the limitations of existing systems.

Original authors: MOSI. AI, :, Donghua Yu, Zhengyuan Lin, Hanfu Chen, Chen Yang, Yiyang Zhang, Jingqi Chen, Ke Chen, Liwei Fan, Yi Jiang, Jie Zhu, Muchen Li, Wenxuan Wang, Yang Wang, Zhe Xu, Botian Jiang, Yitian Gong

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: MOSI. AI, :, Donghua Yu, Zhengyuan Lin, Hanfu Chen, Chen Yang, Yiyang Zhang, Jingqi Chen, Ke Chen, Liwei Fan, Yi Jiang, Jie Zhu, Muchen Li, Wenxuan Wang, Yang Wang, Zhe Xu, Botian Jiang, Yitian Gong, Yuqian Zhang, Wenbo Zhang, Songlin Wang, Zhiyu Wu, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting in a crowded coffee shop, trying to write down exactly what three different friends are saying to each other. You need to know not just the words, but who said them and exactly when they spoke. This is the challenge of "Speaker-Attributed, Time-Stamped Transcription" (SATS). It's a superpower for computers, turning a messy audio recording into a clean, organized script where every sentence is tagged with a speaker's name and a precise clock time. This is incredibly useful for things like transcribing business meetings, analyzing customer service calls, or helping people with hearing impairments follow complex conversations.

Until now, computers usually tried to solve this in two separate steps. First, a "speech-to-text" robot would listen and type out the words. Then, a different "speaker detective" robot would try to figure out who said what. The problem is that these two robots often don't talk to each other. If the first robot makes a small mistake, the second robot gets confused, and the final script becomes a jumbled mess. It's like trying to assemble a puzzle where half the pieces are from a different box. This new paper introduces a new kind of computer brain that does both jobs at the same time, in one single, smooth motion.

The One-Brain Solution

The team behind this project, OpenMOSS, has built a new model called MOSS Transcribe Diarize. Think of this model as a super-intelligent, multi-tasking conductor. Instead of hiring a separate orchestra for the words and a separate one for the voices, this conductor holds the entire score in their head and directs the whole performance in one go.

In the past, computers struggled with long conversations. If a meeting lasted an hour, the old systems would have to chop the audio into tiny 30-second slices, solve each slice, and then try to glue them back together. This often led to the computer forgetting who "Speaker A" was after the break, or losing track of the conversation's flow. MOSS Transcribe Diarize changes the game by having a massive "memory window" of 128k tokens. To put that in perspective, this allows the model to listen to and understand up to 90 minutes of audio in a single, uninterrupted pass. It doesn't need to chop the audio up; it hears the whole story from start to finish, keeping a clear mental picture of who is who, even if they haven't spoken in a while.

How It Works (The Magic Trick)

The model is a "multimodal large language model," which is a fancy way of saying it can read text and listen to audio at the same time. Here is the trick:

The Ears: It takes the raw sound waves and converts them into a digital format.
The Translator: It uses a special "projection" to turn those sound waves into a language the text-reading part of the brain understands.
The Output: Instead of just spitting out words, it outputs a script that looks like this: [0.11] [S01] Good morning! [1.03] [S02] Morning, guys!. It tells you the speaker ID (S01, S02), the exact time they started speaking, and the words they said, all in one single forward pass.

To teach this model, the researchers didn't just use clean, studio-quality recordings. They fed it a massive diet of "real wild" data—recordings from the internet that include background noise, overlapping voices, different accents, and people talking over each other. They also created "simulated" conversations where they mixed and matched different voices to teach the model how to handle tricky situations, like when two people talk at the exact same time.

What They Found

The team tested their new model against some of the biggest, most expensive commercial systems currently available (like those from Doubao, ElevenLabs, and various Google models). They used three different types of tests:

AISHELL-4: Long, real-world meeting recordings (up to ~40 minutes).
Podcast: High-quality, long interviews with multiple guests.
Movies: Short clips with fast, overlapping dialogue.

The results were impressive. In almost every test, MOSS Transcribe Diarize made fewer mistakes than the competition.

Accuracy: It got the words right more often (lower Character Error Rate).
Identity: It was much better at keeping track of who said what. The researchers measured this with a metric called Δcp (the difference between word errors and speaker errors). A lower number here means the model didn't get confused about who was speaking. MOSS Transcribe Diarize had the lowest Δcp scores, meaning it kept the speaker identities consistent even in long, messy conversations.
Long-Form Capability: While some of the famous commercial models (like GPT-4o and Gemini 3 Pro) simply failed to process the long audio files or couldn't output the correct format for them, MOSS Transcribe Diarize handled the full 90-minute inputs without breaking a sweat.

Why This Matters

The paper suggests that by combining speech recognition and speaker identification into one unified system with a long memory, we can get much more reliable transcripts. The model proves that you don't need to chop up audio to understand it; you can listen to the whole conversation at once and still get the details right.

The authors are careful to note that while their model is a significant step forward, it is not a magic wand that solves every problem perfectly. They still see room for improvement, such as making the system work in real-time (streaming) and handling even more languages. But for now, they have shown that a single, unified brain can do the job of two separate robots, and do it better, especially when the conversation gets long and complicated.

The code and the model are now open for anyone to try, available on GitHub and Hugging Face, inviting others to see if they can build even better tools on top of this new foundation.

Technical Summary: MOSS Transcribe Diarize

Problem Statement

Speaker-Attributed, Time-Stamped Transcription (SATS) is a critical task for applications ranging from meeting assistants to legal discovery. It requires not only transcribing speech but also precisely determining who spoke when, preserving turn structure, overlaps, and long-range references.

Current SATS systems typically rely on modular, cascaded pipelines that stitch together Automatic Speech Recognition (ASR) and Speaker Diarization (SD) components. These approaches suffer from several limitations:

Error Propagation: Mistakes in early stages (e.g., ASR) cascade through subsequent stages (e.g., diarization), and global context is difficult to leverage consistently.
Limited Context: Many existing models are constrained by short context windows, forcing long recordings to be processed in chunks. This disrupts discourse continuity, weakens coreference resolution, and introduces boundary artifacts.
Speaker Memory Drift: Speaker identities often drift over long conversations, particularly under varying acoustic conditions or with similar voices.
Lack of Native Timestamps: Many architectures cannot natively output segment-level timestamps, relying on external alignment components that reintroduce modular error propagation.

While recent Multimodal Large Language Models (MLLMs) have shown promise, they often fall short in SATS settings by being limited to short audio contexts (50–90 seconds), small speaker sets, or failing to natively output explicit, time-stamped speaker segments.

Methodology

The authors present MOSS Transcribe Diarize, a unified multimodal large language model designed to perform SATS in a single, end-to-end pass.

Architecture

The model couples an audio encoder with a projection module that maps multi-speaker acoustic embeddings into the feature space of a pretrained text LLM.

Unified Modeling: This architecture allows the backbone to jointly align speaker identities with lexical content and perform unified, long-context modeling.
Explicit Timestamping: Following recent work on textual token-based time encoding, temporal information is represented as formatted timestamp text inserted between audio encoder chunks. This avoids binding temporal encoding to absolute positional indices, which become sparse over long durations, enabling accurate timestamp generation over hour-scale audio.
Context Window: The model is equipped with a 128k-token context window, allowing it to process inputs up to 90 minutes without chunking. This preserves discourse continuity and maintains global representations of participants and topics.

Data Composition

The model is trained on extensive real-world conversational audio and simulated data to ensure robustness:

Real Data: Includes multilingual audio from the internet, specifically the AISHELL-4 dataset (meeting room recordings with far-field overlapping audio), as well as curated podcast and film datasets.
Simulated Data: To address the scarcity of high-quality real-world multi-speaker recordings, the authors employ a controllable probabilistic simulator. This generates synthetic multi-speaker dialogues with 2–12 speakers, enforcing speaker alternation while permitting overlaps (capped at 80% of the shorter segment). The mixtures are augmented with real-world noise and reverberation (SNRs from 0–15 dB) to improve perceptual continuity.

Key Contributions

End-to-End SATS System: MOSS Transcribe Diarize is presented as the first unified multimodal model to jointly perform word recognition, speaker attribution, and timestamp prediction in a single forward pass, eliminating hand-offs between separate subsystems.
Meeting-Scale Long-Context Modeling: The model processes inputs up to 90 minutes within a 128k-token context window. This capability reduces identity drift and boundary artifacts, improving SATS metrics across both in-domain and out-of-domain evaluations without the need for chunking.

Evaluation and Results

The model was evaluated against state-of-the-art commercial systems (including Doubao, ElevenLabs, GPT-4o, and Gemini variants) across three diverse benchmarks:

AISHELL-4: Long-form (nearly 40-minute) real-world conference recordings.
Podcast: High-quality, long-duration, multi-guest interviews.
Movies: Short, overlap-rich segments from films and TV series.

Metrics:
Performance was measured using Character Error Rate (CER) for ASR, concatenated minimum-permutation CER (cpCER) for joint ASR and SD, and $\Delta_{cp}$ (the difference between cpCER and CER) to isolate speaker attribution errors.

Findings:

Superior Joint Performance: MOSS Transcribe Diarize consistently achieved the best overall performance in terms of cpCER and $\Delta_{cp}$ across all datasets.
Reduced Attribution Error: On the AISHELL-4 dataset, the model exhibited a markedly smaller $\Delta_{cp}$ compared to baselines, indicating that speaker attribution errors introduced significantly less additional degradation than pure ASR errors.
Long-Form Capability: Unlike GPT-4o and Gemini 3 Pro, which were unable to reliably process long-form audio inputs under the evaluation protocol due to input length constraints or format instability, MOSS Transcribe Diarize remained fully operational on hour-scale audio.
Robustness to Overlaps: In the Movies dataset (characterized by rapid speaker alternation and frequent overlaps), the model maintained a relatively small gap between CER and cpCER, demonstrating robust handling of speaker boundaries.

Significance

The paper claims that MOSS Transcribe Diarize addresses the brittleness of modular ASR–diarization pipelines by unifying recognition and speaker attribution within a single multimodal framework. By leveraging long-context modeling and distribution-controlled simulation, the system achieves more reliable speaker-attributed, time-stamped transcripts than modular or semi-cascaded alternatives. The authors emphasize that this approach bridges the gap between nominal multimodal capability and deployable long-form SATS performance, making it particularly well-suited for real-world meeting transcription and long-form conversational analysis.