💬 NLP

Investigating Knowledge Transfer Across Interactive Dialogue Games

This paper investigates knowledge transfer across interactive dialogue games by analyzing finetuned LLMs on the Clembench suite, revealing that visuospatial exploration games transfer best while similarity-based task vectors fail to capture complex transferability patterns.

Original authors: Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda, Pavel Merinov, Raffaella Bernardi, Oswald Lanz, Alessandro Suglia, Alessandro Torcinovich

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Filippo Momentè, Mir Nafis Sharear Shopnil, Andrea de Varda, Pavel Merinov, Raffaella Bernardi, Oswald Lanz, Alessandro Suglia, Alessandro Torcinovich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of artificial intelligence, researchers have long relied on static tests to measure how well a computer understands language. These tests are like multiple-choice exams, where a model selects the right answer from a list. But real conversation is rarely a test; it is a dynamic, back-and-forth exchange where people must work together to solve problems, follow rules, and achieve goals. To bridge this gap, scientists have turned to "dialogue games." These are structured scenarios where an artificial intelligence must use language to navigate a situation, such as guessing a word based on clues or guiding a partner through a virtual maze. The central question driving recent research is not just whether a model can win a single game, but whether the skills it learns in one game can help it win others. If a machine learns to navigate a complex maze, does that make it better at guessing words? Or are these skills entirely separate?

A team of researchers set out to map these connections by treating dialogue games as a laboratory for learning. They began with a collection of seventeen different games, ranging from word-based puzzles like "Taboo," where players must describe a word without using forbidden terms, to spatial challenges like "Adventure Game," where a player must explore a virtual environment and move objects to reach a goal. They divided these games into two broad families: those that rely on verbal reasoning and those that require visuospatial thinking, such as understanding grids and physical layouts. The researchers then took a large language model and trained it specifically on each of these games, one by one. This process created a series of specialized experts, each an expert in a single game but potentially carrying hidden knowledge that could help with the others.

To see how this knowledge moved between games, the team ran a massive series of experiments. They took a model trained on one game and tested it on all the others to see if its performance improved. They discovered that the transfer of knowledge was not random; it followed a clear pattern. The most surprising finding was that the games involving spatial reasoning were the most generous teachers. A model trained to navigate a virtual world or draw a grid based on instructions showed a remarkable ability to improve its performance on purely verbal games. For instance, the skills learned in an exploration game helped a model play "Taboo" better, even though the two games seem completely different on the surface. In contrast, training on a verbal game rarely helped a model with a spatial task. The researchers found that the spatial family of games provided a kind of universal foundation, teaching the model versatile skills like planning and strategy that could be applied anywhere.

The team also looked for a simpler way to predict these results without having to play the games over and over. They examined the internal "weights" of the models—the mathematical adjustments made during training—to see if games that helped each other looked similar inside the computer's brain. They hoped that if two games were related, the changes to the model's internal structure would be similar, like two fingerprints matching. However, this approach failed to predict the results. They found that while models trained on the same game with different roles (like a clue-giver versus a guesser) did look similar internally, this similarity did not tell them whether one game would help the other. Two games could look very similar inside the model's brain but still offer no help to each other when played. Conversely, games that looked quite different internally could still provide a massive boost in performance.

This suggests that the true measure of how one game helps another is not found in the static structure of the model, but in the act of playing. The researchers concluded that while we can see what a model has learned by looking at its internal changes, we cannot predict how that learning will transfer to a new task without actually testing it. The study highlights that dialogue games require complex, layered skills that are instantiated in different ways. Some games, particularly those involving spatial reasoning, act as powerful training grounds that build a flexible intelligence capable of tackling a wide variety of challenges. Others are more specialized, offering little help outside their own narrow domain. By mapping these relationships, the researchers have provided a clearer picture of how artificial intelligence learns to think, showing that the path to a more capable model may lie in teaching it to navigate the world, not just to speak about it.

Technical Summary: Investigating Knowledge Transfer Across Interactive Dialogue Games

Problem Statement
Dialogue games represent a complex setting for evaluating Large Language Models (LLMs), requiring the coordination of cognitive skills, rule-following, and strategic reasoning to achieve goals through multi-turn interaction. While prior work has utilized dialogue games to reveal capabilities missed by static benchmarks (e.g., MMLU), these studies typically treat game performance as an end goal rather than a lens to understand the internal organization of model capabilities. There is currently a lack of systematic investigation into how knowledge transfers across different dialogue games. Specifically, it remains unclear whether training a model on one specific game enhances its ability to play others, and if so, what structural relationships govern these transfer effects.

Methodology
The authors investigate knowledge transferability using the clembench evaluation framework (Chalamalasetti et al., 2023), focusing on 15 role-specific tasks derived from 9 textual and multimodal dialogue games. These games are categorized into two families: verbal (operating on words, e.g., Codenames, Taboo) and visuospatial (operating on grids and spatial configurations, e.g., Adventure Game, Image Game).

The study employs two complementary analytical approaches:

Task-Transferability Graph (Taxonomy Modeling):
- Data Preparation: The authors fine-tuned a baseline LLM (Qwen3.5-4B) on specific game transcripts to create "task specialists." Each specialist was evaluated on all tasks using the clembench quality score (qs), which disentangles game-playing competence from mere rule-following.
- Optimization: To map transfer relationships, the authors adapted the Taskonomy framework (Zamir et al., 2018). They formulated a Collective Transferability Problem (CTP) as a binary integer optimization program. The objective is to select a subset of source tasks (within a budget $\gamma$ ) that maximizes the collective performance on all target tasks.
- Metric: Transferability is defined as the performance improvement on a target task $t$ when a model is trained on a source task $s$ , relative to the baseline.
Weight Space Analysis:
- Task Vectors: The authors computed task vectors (Ilharco et al., 2022), defined as the difference between the parameters of a fine-tuned specialist ( $\theta_t$ ) and the baseline model ( $\theta_{Base}$ ).
- Similarity Metrics: They analyzed the geometry of these vectors using two metrics:
  - Cosine Similarity: A global measure of vector alignment.
  - Subspace Overlap: A layer-wise metric calculating the overlap of the top- $k$ singular vector subspaces of the weight updates (specifically in the MLP blocks of the transformer).
- Goal: To determine if weight-space similarity can predict or explain the transfer relationships observed in the gameplay performance.

Key Results

Visuospatial Superiority: The transferability analysis reveals that visuospatial games (e.g., exploration and grid-based tasks) transfer knowledge significantly better than verbal games. Specifically, adventuregame acts as a robust baseline specialist, providing transferable skills (such as planning and navigation) that benefit purely verbal games like Taboo and Codenames.
Asymmetry and Non-Linearity: Transfer relationships are often asymmetric. The study identifies cases where fine-tuning on Game A improves performance on Game B, but fine-tuning on Game B degrades performance on Game A. Furthermore, the authors observe that fine-tuning on a specific game is not always the optimal strategy for improving at that same game; in some instances, training on a different game yields better results.
Limitations of Weight Similarity: The weight-space analysis shows that while task vectors for different roles within the same game are highly aligned (high subspace overlap), this similarity does not correlate with transferability.
- High similarity (e.g., between Image Game roles) can result in either mutual help or mutual harm.
- Asymmetric transfers (where one direction helps and the other hurts) often exhibit low similarity.
- The authors conclude that weight similarity acts as a "fingerprint" of the game and player role rather than a predictor of transferability.

Key Contributions

Systematic Transfer Analysis: This work provides the first systematic study of knowledge transfer across a broad suite of dialogue games, moving beyond isolated task evaluations.
Taxonomy of Dialogue Games: By applying the Taskonomy framework to NLP, the authors construct a graph representation of transfer relationships, identifying the visuospatial family as a source of versatile, generalizable skills.
Decoupling Similarity from Transfer: The paper demonstrates that parameter-space similarity (task vectors) is insufficient for explaining transferability. It highlights a distinction between "what was trained" (captured by weight updates) and "how that training generalizes" (captured only by gameplay evaluation).
Methodological Adaptation: The successful adaptation of computer vision transfer frameworks (Taskonomy) and mechanistic interpretability tools (Task Vectors) to the domain of interactive dialogue games.

Significance and Claims
The authors claim that their work offers an "interpretability handle" on the effects of game-specific fine-tuning. By showing that visuospatial reasoning provides a foundation for broader dialogue capabilities, the study suggests that the cognitive demands of spatial reasoning may be more fundamental to complex dialogue interaction than previously assumed.

The paper modestly frames its findings as exploratory. It acknowledges limitations, including the focus on collaborative games within the clembench suite and the exclusion of negative transfer analysis. The authors posit that their results support the view that dialogue games require complex, instantiated natural language understanding and reasoning skills. They suggest that future work should explore non-symmetrical operators to better predict transferability without requiring evaluation on target games and investigate curriculum learning regimes based on these identified transfer relationships.

Technical Summary: Investigating Knowledge Transfer Across Interactive Dialogue Games

More like this