💬 NLP

A Generative Model for Joint Multiple Intent Detection and Slot Filling

This paper proposes a generative framework featuring an attention-over-attention decoder to simultaneously handle multiple intent detection and slot filling, achieving state-of-the-art performance on public and newly constructed multi-intent datasets.

Original authors: Liz Li, Wei Zhu

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liz Li, Wei Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, but slightly rigid, robot assistant.

The Problem: The "One-Task-at-a-Time" Robot
Most current robot assistants are trained to handle only one request at a time. If you say, "Play some jazz and order a pizza," the robot gets confused. It might try to play jazz and order a pizza, but it treats them as two separate, unrelated events, or it might get the order wrong because it's trying to force everything into a single box.

In the real world, humans are messy. We often say things like, "Find me a flight to Paris and then book a hotel near the Eiffel Tower." We have multiple intents (finding a flight, booking a hotel) and multiple details (Paris, Eiffel Tower) all in one breath. Existing robots struggle with this "multi-task" life.

The Solution: The "Creative Storyteller" (GEMIS)
The authors of this paper built a new robot brain called GEMIS. Instead of trying to solve the problem by checking boxes (like a checklist), they taught the robot to tell a story.

Here is how they did it, using some simple analogies:

1. The Shift: From "Filing Cabinets" to "Writing a Recipe"

Old Way (The Filing Cabinet): Traditional models try to file your sentence into two separate drawers: one drawer for "What do you want?" (Intent) and another for "What are the details?" (Slots). If you have two requests, the drawers get messy and the robot forgets how they connect.
New Way (The Recipe): GEMIS treats your sentence like a recipe. It doesn't just file things away; it writes out a structured story of what you want.
- Input: "Play jazz and order pizza."
- Output Story: "Intent: Play Music. Slot: Jazz. Intent: Order Food. Slot: Pizza."
- By writing it out as a sequence, the robot naturally understands that the "Jazz" belongs to the "Play Music" part, and "Pizza" belongs to the "Order Food" part. It flows naturally, just like reading a sentence.

2. The Secret Sauce: "Attention-over-Attention" (The Spotlight)

The robot uses a special tool called BART, which is like a super-smart reader that has read almost everything on the internet. But the authors added a special upgrade called Attention-over-Attention (AoA).

The Analogy: Imagine you are directing a play.
- Standard Attention: The director points a spotlight at the actor speaking right now.
- Attention-over-Attention: The director looks at where the spotlight was in the previous scene to decide where to point it next.
- Why it matters: If the robot just heard "Play Music," the "spotlight" knows to look for music details next. If it heard "Book Hotel," the spotlight shifts to look for dates and locations. This "spotlight memory" helps the robot keep track of multiple requests without getting confused, even if they are complex.

3. The Training Data: "The Natural Conversation"

To teach this robot, you need good examples.

The Old Problem: Previous researchers made up fake examples by randomly gluing sentences together, like: "I want to fly to Tokyo. Also, what is the weather in Antarctica?" This is unnatural. Real people don't talk like that.
The New Solution: The authors used a "Coherence Detector" (a part of a model called BERT). They asked the detector: "Do these two sentences sound like they belong together in a real conversation?"
- If you say, "Fly to Tokyo" and "Book a hotel in Tokyo," the detector says, "Yes, these go together!"
- If you say, "Fly to Tokyo" and "What is the weather in Antarctica?" the detector says, "No, that's weird."
- They only kept the "Yes" examples to build a new, high-quality training dataset. This taught the robot to understand how real humans actually chain their thoughts together.

The Result: The Super-Listener

When they tested this new robot:

It understood complex requests much better than the old robots.
It got smarter the more requests you gave it. While other robots crashed when you asked for three things at once, this one handled it with ease.
It worked on both fake and real data. It proved that by teaching the robot to "write a story" of your request rather than "fill out a form," it becomes a much better listener.

In a nutshell:
The paper says, "Stop trying to force human conversation into rigid boxes. Instead, teach the AI to read your sentence like a story, use a 'spotlight' to remember what it just said, and only learn from conversations that sound like real humans talking." The result is a robot that finally understands when you say, "Do this, and then do that."

1. Problem Statement

In task-oriented dialogue systems, Spoken Language Understanding (SLU) is critical and comprises two sub-tasks: Intent Detection (classifying the user's goal) and Slot Filling (extracting specific entities).

The Challenge: Most existing research and datasets focus on single-intent scenarios. However, in real-world interactions, users frequently express multiple intents within a single utterance (e.g., "Play music and check the weather").
Limitations of Current Approaches:
- Modeling: Existing multi-intent models often rely on complex interaction modules (e.g., Graph Neural Networks) that are trained from scratch, failing to fully leverage the power of pre-trained language models (PLMs).
- Data: Existing multi-intent datasets (e.g., MixATIS, MixSNIPS) are constructed by randomly concatenating single-intent utterances. This results in unnatural samples where semantically unrelated intents appear together, which does not reflect real-world user behavior.

2. Methodology: GEMIS

The authors propose GEMIS (GEnerative framework for Multi-Intent SLU), a unified generative approach that reformulates the joint task as a Sequence-to-Sequence (Seq2Seq) problem.

A. Generative Framework Reformulation

Instead of treating intent detection as classification and slot filling as sequence labeling separately, GEMIS converts the entire task into a generation task:

Input: The original user utterance ( $X$ ).
Output: A structured target sequence ( $Y$ $Y$ ) containing:
1. All detected intents.
2. Slot triplets defined as $(start\_index, end\_index, slot\_category)$ .
Advantage: This allows the model to naturally handle a variable number of intents and slots using an auto-regressive decoder, capturing the dependency where slot filling can condition on predicted intents.

B. Model Architecture (Modified BART)

The model is built upon the pre-trained BART (Bidirectional and Auto-Regressive Transformers) model, with specific modifications:

Pointer Network: Since the target sequence includes position indices (start/end of slots) which are not part of the standard BART vocabulary, a Pointer Network is added on top of the decoder. This allows the model to point to specific tokens in the input sequence to identify slot boundaries.
Attention-over-Attention (AoA): This is the core innovation.
- Problem: Standard cross-attention in BART decoders may not sufficiently capture the specific interaction patterns between intent predictions and slot generation.
- Solution: The authors replace standard cross-attention layers with an AoA structure.
- Mechanism: AoA utilizes the Cross-Attention Maps (CAMs) from previously generated labels (intents/slots) as a guide. It calculates a self-attention map (SAM) over the current step and uses it to weigh the historical CAMs. This creates a new attention map that incorporates "inductive bias," effectively guiding the generation of current slots based on the attention patterns of previously predicted intents.

C. Dataset Construction (Multi-ATIS & Multi-SNIPS)

To address the lack of natural multi-intent data, the authors constructed two new datasets:

Method: Instead of random concatenation, they utilized the Next Sentence Prediction (NSP) head of a pre-trained BERT model.
Process: They iteratively concatenated single-intent utterances only if the NSP probability (coherence score) exceeded a specific threshold.
Result: This ensures that the combined intents are semantically related and likely to co-occur in real conversations (e.g., "Add to playlist" and "Play music"), resulting in Multi-ATIS and Multi-SNIPS.

3. Key Contributions

Generative Framework: Proposed GEMIS, the first generative framework to solve joint multiple intent detection and slot filling as a unified Seq2Seq task, avoiding the need for separate task-specific heads.
Attention-over-Attention (AoA): Designed a novel decoder module that leverages historical attention patterns to guide slot filling, significantly improving the interaction between the two sub-tasks.
High-Quality Datasets: Constructed Multi-ATIS and Multi-SNIPS using BERT's NSP head, providing more realistic and coherent multi-intent samples compared to previous random-concatenation datasets.
State-of-the-Art Performance: Demonstrated that the proposed approach outperforms existing methods across various metrics.

4. Experimental Results

The model was evaluated on four datasets: the public MixATIS/MixSNIPS and the newly constructed Multi-ATIS/Multi-SNIPS.

Performance on Public Datasets (MixATIS/MixSNIPS):
- GEMIS achieved 53.4% overall accuracy on MixATIS and 87.4% on MixSNIPS.
- This significantly outperformed the previous best baselines (GL-GIN), which scored 43.5% and 75.4% respectively.
- The AoA module alone contributed a 2.3% absolute improvement in overall accuracy on MixSNIPS over the vanilla generative model.
Performance on Constructed Datasets (Multi-ATIS/Multi-SNIPS):
- GEMIS showed even larger gains here, improving overall accuracy by 11.3% (Multi-ATIS) and 17.6% (Multi-SNIPS) over baselines.
- This suggests GEMIS is particularly effective in handling natural, coherent multi-intent scenarios.
Impact of Intent Count:
- As the number of intents per utterance increases (from 1 to 3), the performance of all models degrades.
- However, GEMIS degrades less than baselines. The performance gap between GEMIS and baselines widens as the number of intents increases, proving its robustness in complex scenarios.
Model Size:
- Experiments with BART-LARGE vs. BART-BASE confirmed that larger pre-trained models yield better results, but even the smaller BART-BASE version of GEMIS significantly outperformed previous state-of-the-art models.

5. Significance

This paper represents a paradigm shift in SLU research by moving away from discriminative, multi-head architectures toward a unified generative approach.

Efficiency: It leverages pre-trained parameters (BART) effectively, avoiding the need to train complex interaction modules from scratch.
Realism: By addressing the data quality issue through NSP-based dataset construction, it bridges the gap between synthetic benchmarks and real-world user behavior.
Scalability: The generative nature of GEMIS makes it inherently scalable to handle varying numbers of intents without architectural changes, making it a strong candidate for next-generation task-oriented dialogue systems.