💬 NLP

Pantagruel: Unified Self-Supervised Encoders for French Text and Speech

The paper introduces Pantagruel, a family of unified self-supervised encoder models for French text and speech that learns contextualized feature-space representations to achieve competitive or superior performance across diverse downstream tasks compared to existing French baselines.

Original authors: Phuong-Hang Le, Valentin Pelloin, Arnault Chatelain, Maryem Bouziane, Mohammed Ghennai, Qianwen Guan, Kirill Milintsevich, Salima Mdhaffar, Aidan Mannion, Nils Defauw, Shuyue Gu, Alexandre Audibert, M

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Phuong-Hang Le, Valentin Pelloin, Arnault Chatelain, Maryem Bouziane, Mohammed Ghennai, Qianwen Guan, Kirill Milintsevich, Salima Mdhaffar, Aidan Mannion, Nils Defauw, Shuyue Gu, Alexandre Audibert, Marco Dinarelli, Yannick Estève, Lorraine Goeuriot, Steffen Lalande, Nicolas Hervé, Maximin Coavoux, François Portet, Étienne Ollion, Marie Candito, Maxime Peyrard, Solange Rossato, Benjamin Lecouteux, Aurélie Nardy, Gilles Sérasset, Vincent Segonne, Solène Evain, Diandra Fabre, Didier Schwab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the French language. In the past, you had to teach it two completely different subjects: Reading (text) and Listening (speech). You'd use one teacher for books and a totally different teacher for radio shows. They spoke different languages, used different textbooks, and rarely talked to each other.

Enter Pantagruel. Think of Pantagruel not as a robot, but as a super-smart, bilingual Swiss Army Knife designed specifically for French. It's a new family of AI models that learns French text and French speech using the same brain architecture, just tuned slightly differently for each job.

Here is the breakdown of how it works, using some everyday analogies:

1. The Old Way vs. The New Way

The Old Way (Token Reconstruction):
Imagine trying to learn a language by playing "Fill in the Blank." You show the student a sentence with a missing word, and they have to guess the exact word that fits.

Problem: This is great for text (guessing "cat" vs. "dog"), but it's terrible for speech. Speech is a continuous wave of sound, like a river. You can't really "fill in the blank" in a river; you need to understand the flow, the rhythm, and the emotion.

The Pantagruel Way (Feature-Space Prediction):
Instead of guessing the exact word or sound, Pantagruel plays a game of "Guess the Vibe."

The Analogy: Imagine you are looking at a painting with a piece of tape covering a corner.
- Old Method: You have to guess the exact color of the paint under the tape.
- Pantagruel Method: You have to guess the feeling or the concept of what's under the tape. Is it a sunny landscape? A stormy sea? A portrait?
Why it's better: This allows the AI to understand the deep structure of the language (the "vibe") rather than just memorizing the surface details. It works beautifully for both the "river" of speech and the "blocks" of text.

2. The Secret Ingredient: The "Teacher-Student" Game

Pantagruel learns using a Teacher-Student system, which is like a master chef teaching an apprentice.

The Teacher: The master chef sees the entire dish (the full sentence or audio clip). They know exactly what the final flavor should be.
The Student: The apprentice sees the dish with a few ingredients hidden (masked).
The Lesson: The student tries to guess what the hidden ingredients would taste like based on the rest of the dish. The teacher doesn't give the answer; they just give a "flavor profile" (a mathematical representation) of what the hidden part should be.
The Twist: The teacher is actually a "ghost" version of the student that moves slowly and steadily (an "Exponential Moving Average"). This prevents the student from getting confused by their own mistakes and helps them learn a stable, robust understanding of French.

3. The Massive Library (The Data)

To become an expert, you need to read and listen to everything. Pantagruel was fed a massive diet of French data:

For Reading: It ate through Wikipedia, the entire internet (OSCAR), and a specialized library called CroissantLLM.
For Listening: This is where Pantagruel shines. It was trained on 100,000 hours of French audio from the INA (the French National Audiovisual Institute).
- The Analogy: Most AI models have listened to a few hours of audiobooks. Pantagruel has listened to every radio show, TV news broadcast, documentary, and talk show in France for decades. It has heard French spoken by grandmothers in villages, news anchors in Paris, children playing, and people shouting in markets. This makes it incredibly tough and adaptable to noise and different accents.

4. The Hybrid Approach for Text

The researchers realized that while "Guessing the Vibe" is amazing for speech, text is a bit different. Text is made of discrete blocks (words) that have very specific grammar rules.

The Solution: They gave Pantagruel a hybrid diet. For speech, it only "Guesses the Vibe." For text, it does that plus the traditional "Fill in the Blank" game. This ensures it understands both the deep meaning and the strict grammar rules of French.

5. The Results: Why Should You Care?

When they tested Pantagruel on a wide variety of tasks—like translating speech, understanding medical prescriptions, recognizing emotions in phone calls, or answering questions—it beat or matched all the previous best French models.

The Takeaway: Pantagruel proves that you don't need two separate brains for reading and listening. With the right training method (predicting the "vibe" rather than just the "word") and a massive, diverse diet of data, you can build one unified system that understands French in all its forms.

In a nutshell: Pantagruel is the first AI to truly "speak" and "read" French with the same deep, intuitive understanding, thanks to a clever learning game and a massive library of French radio and TV history. It's a giant leap toward making AI that truly understands human communication, not just the words on the page.

1. Problem Statement

While self-supervised learning (SSL) has revolutionized French language processing, existing models face two primary limitations:

Modality Fragmentation: Text models (e.g., CamemBERT, FlauBERT) and speech models (e.g., LeBenchmark) are typically trained separately using modality-specific objectives. Text models rely on token-level reconstruction (Masked Language Modeling - MLM), while speech models often use contrastive learning or discrete unit prediction. This hinders a unified approach to multimodal understanding.
Limitations of Token-Level Objectives: Traditional token-level reconstruction (predicting the next token) may under-utilize the structural regularities of continuous signals (speech) and can be suboptimal for capturing deep contextual representations in a unified feature space.
Data Scarcity in French: While English has massive datasets (e.g., CommonVoice, LibriSpeech), French lacks a comparable, diverse, large-scale broadcast corpus for pre-training speech models, limiting robustness to real-world acoustic variability.

2. Methodology

The authors introduce Pantagruel, a family of self-supervised encoder models for French text and speech, built upon the data2vec 2.0 architecture, an instance of the Joint Embedding Predictive Architecture (JEPA) framework.

A. Architecture and Training Paradigm

Teacher-Student Setup: The model employs a teacher-student paradigm. The Teacher observes the full, unmasked input and produces latent representations. The Student observes a masked version of the input and attempts to predict the Teacher's latent representations for the masked regions.
Feature-Space Prediction: Instead of reconstructing raw inputs or discrete tokens, the model minimizes the L2 distance between the student's predicted features and the teacher's target features in the embedding space.
Unified Framework: The same architecture (Transformer encoder + lightweight decoder) is used for both modalities, differing only in the input pre-net (CNN for audio, embedding layer for text).

B. Modality-Specific Adaptations

Speech (Pure JEPA): For speech, the model uses a pure feature-space prediction objective (masked feature prediction). This allows the model to capture rich acoustic and linguistic regularities without being constrained by discrete tokenization.
Text (Hybrid Objective): The authors hypothesized that pure feature-space prediction might struggle with the discrete, sparse nature of text tokens compared to continuous signals. To address this, they propose a hybrid loss function:
$L_{text} = L_{L2} + \lambda L_{MLM}$
Where $L_{L2}$ is the feature-space prediction loss and $L_{MLM}$ is the standard Masked Language Modeling loss. The weight $\lambda$ decays linearly during training. This combines the benefits of contextualized target prediction with fine-grained syntactic/semantic supervision from token-level reconstruction.

C. Data Resources

Text: Pre-trained on massive French corpora including Wikipedia, OSCAR (web crawl), and CroissantLLM (a bilingual French-English dataset).
Speech: Pre-trained on Multilingual LibriSpeech and LeBenchmark datasets. Crucially, the authors introduce INA-100k, a new 100,000-hour corpus derived from the archives of the Institut National de l'Audiovisuel (INA). This dataset includes diverse broadcast content (news, documentaries, ads, etc.) from 1940–2022, significantly expanding the scale and diversity of French speech data available for pre-training.

3. Key Contributions

Pantagruel Model Family: Release of unified French SSL encoders for both text and speech based on the data2vec 2.0/JEPA framework, enabling a shared architecture for multimodal tasks.
Hybrid Text Objective: Investigation of embedding-based prediction for French text, demonstrating that combining JEPA-style feature prediction with MLM yields competitive results, addressing the under-explored regime of non-token-level objectives for text.
INA-100k Corpus: Introduction of a massive, diverse 100k-hour French speech corpus, addressing a critical data gap in French speech research.
Unified Evaluation: A comprehensive benchmarking suite covering both modalities, showing that Pantagruel models can handle text and speech inputs seamlessly.

4. Results

The models were evaluated against strong French baselines (CamemBERT, FlauBERT, LeBenchmark 2.0) across various downstream tasks.

Text Performance

Semantic Tasks: Pantagruel models (specifically those trained on OSCAR and CroissantLLM) achieved competitive or superior performance on semantic tasks like Text Classification, Paraphrase Identification, and Extractive Question Answering (PIAF).
Syntax-Sensitive Tasks: The models showed slightly lower performance on syntax-heavy tasks like Verb Sense Disambiguation (VSD) and Dependency Parsing compared to pure MLM models (CamemBERT). This suggests that while feature-space prediction captures high-level semantics well, token-level supervision remains valuable for fine-grained syntactic structure.
Hybrid Success: The hybrid models (MLM + JEPA) generally outperformed pure JEPA text models, confirming the necessity of the auxiliary MLM loss for French text.

Speech Performance

Superiority over Baselines: Pantagruel models consistently outperformed LeBenchmark baselines across all speech tasks.
Robustness: The Pantagruel-L-114k model (trained on LeBenchmark + INA-100k) achieved state-of-the-art results on challenging, noisy, and spontaneous speech datasets like ETAPE (broadcast news) and DyLNet (child speech).
Cross-Lingual Translation: In Speech-to-Text (ST) translation tasks, Pantagruel models showed significant gains, particularly in low-resource settings, highlighting the effectiveness of latent-based objectives for acoustic representation.
Data Impact: While increasing data from 14k to 114k hours improved performance on diverse broadcast data, it showed a slight trade-off on clean read-speech datasets (CommonVoice), suggesting the model adapts to the specific acoustic variability of the larger corpus.

5. Significance

Multimodal Foundation: Pantagruel represents a significant step toward truly multimodal AI for French, proving that a single architecture can effectively learn from both text and speech using a unified predictive objective.
Resource Democratization: The release of the models and the INA-100k corpus provides the French research community with high-quality, ethically curated resources, reducing the reliance on English-centric models.
JEPA for Language: The work validates the JEPA framework for French language processing, showing that feature-space prediction is highly effective for continuous signals (speech) and, when hybridized, competitive for discrete signals (text).
Future Directions: The authors note that the architecture is already adapted for joint training on unaligned speech-text pairs, paving the way for future research in cross-modal alignment and multimodal downstream tasks (e.g., spoken QA, automatic subtitling).

In conclusion, Pantagruel establishes a new standard for French representation learning, leveraging massive-scale data and advanced predictive architectures to create robust, unified models for both text and speech.