💬 NLP

Playing with Words, Improving with Rewards: Training Language Models for Creative Association

This paper proposes training Large Language Models on the word-association game Codenames using Reinforcement Learning with Verifiable Rewards to enhance creativity, revealing that while larger models (8B) achieve consistent creative gains with minimal reasoning loss, smaller models (1.7B and 4B) prioritize reasoning precision at the expense of creativity.

Original authors: Vijeta Deshpande, Namrata Shivagunde, Sherin Muckatira, Hadrien Glaude, Mikhail Gronas, Claire Stevenson, Roger Beaty, Anna Rumshisky

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vijeta Deshpande, Namrata Shivagunde, Sherin Muckatira, Hadrien Glaude, Mikhail Gronas, Claire Stevenson, Roger Beaty, Anna Rumshisky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of AI students, each with a different size brain: a small one (1.7B), a medium one (4B), and a large one (8B). The researchers wanted to teach these AI students how to be creative.

The problem is that creativity is like art; it's hard to grade. If you ask a human, "Is this poem creative?" they might say yes, while another says no. This makes it hard to train an AI because the AI needs clear feedback to learn.

The Solution: The Word Game "Codenames"

To fix this, the researchers turned creativity into a game called Codenames.

Think of a board covered in random words like "Apple," "River," "Chair," and "Cloud."

The Goal: One player (the "Spy Master") sees a secret list of target words (e.g., "Apple" and "River"). They must give a single clue word (like "Fruit") that connects to their targets but doesn't accidentally connect to the other words on the board (like "Chair").
The Challenge: This requires two types of thinking:
1. Divergent Thinking: Looking at "Apple" and "River" and thinking, "What do they have in common? Oh, maybe 'Nature' or 'Growth'?" (Stretching the mind).
2. Convergent Thinking: Picking the one best word that fits perfectly without tripping up the team. (Narrowing the focus).

Because the game has a clear win/loss condition (did you guess the right words?), the AI can learn by playing millions of rounds and getting a simple "Good job" or "Try again" score. No human judges needed.

The Experiment: Teaching with Rewards

The researchers used a method called Reinforcement Learning with Verifiable Rewards (RLVR).

Imagine the AI is a dog. Every time it makes a good move in the game, it gets a treat (a reward). Every time it makes a bad move, it gets no treat.
They trained the three AI models (Small, Medium, Large) to play this game over and over until they got really good at it.

The Surprising Discovery: Size Matters

Here is the twist. The researchers expected all the AIs to get better at everything. Instead, they found that the size of the AI's brain changed what it learned.

1. The Small and Medium AIs (1.7B and 4B): The "Precision Specialists"

What happened: These smaller models got significantly better at logic and math problems (like solving puzzles or doing arithmetic).
The Trade-off: They actually got worse at being creative. They became very strict and precise, like a robot accountant. They stopped taking risks and started looking for the "safe" answer.
Analogy: It's like teaching a small child a complex board game. They learn the rules perfectly and stop making mistakes, but they stop playing with their imagination.

2. The Large AI (8B): The "Creative Explorer"

What happened: The biggest model got significantly better at creative tasks (writing stories, coming up with funny captions, making metaphors).
The Trade-off: It got slightly worse at strict math and logic. It started taking more risks and making more unique connections, which sometimes led to small errors in calculation.
Analogy: It's like teaching a brilliant artist the same board game. They learn the rules but start seeing hidden patterns and making wild, creative connections that the smaller players miss.

The Big Picture

The paper concludes that you can't just "turn on" creativity in all AI models the same way.

If you want a precise, logical thinker, training on this word game helps the smaller models become sharper.
If you want a creative, imaginative thinker, training on this game helps the larger models become more diverse and inventive.

The researchers showed that by using a simple, objective game, they could "tune" AI models to be either better at logic or better at creativity, depending on how big the model is. It's a practical way to shape AI behavior without needing humans to constantly grade their work.

Technical Summary: Playing with Words, Improving with Rewards

Problem Statement

Large Language Models (LLMs) are increasingly tasked with solving complex problems requiring creativity, defined as the capacity to navigate vast solution spaces and generate unconventional yet effective solutions. However, training LLMs for creativity is hindered by the subjective nature of the concept; human judgment on what constitutes "creative" work is inconsistent, making standard Reinforcement Learning from Human Feedback (RLHF) difficult to scale and noisy. The central challenge is how to explicitly train LLMs to be genuinely creative without relying on expensive and subjective human preference annotation.

Methodology

The authors propose a framework that reframes creativity as a game with objectively verifiable outcomes, bypassing the need for human judgment.

1. Training Environment: Codenames
The study utilizes Codenames, a word-association game, as the training environment. The game inherently exercises the two central axes of creative cognition:

Divergent Thinking: Generating remote associations spanning multiple concepts.
Convergent Thinking: Selecting the single best association that uniquely identifies targets while avoiding non-targets.

The training setup simplifies the full game into a Markov Decision Process (MDP) consisting of two single-turn tasks:

Clue Generation: Given a set of target words ( $T$ ) and non-target words ( $N$ ), the model generates a single clue word ( $w_{clue}$ ) that connects to as many targets as possible while avoiding associations with non-targets.
Guess Generation: Given a board of words ( $B = T \cup N$ ) and a pre-computed clue, the model guesses which words the clue refers to.

2. Data Synthesis and Curriculum
To ensure tractability and variety, the authors generated a dataset of 3,680 training examples. They created a pool of 1,000 diverse topics and associated words (stratified by frequency into common and rare). The training data follows a curriculum of increasing difficulty:

Simple: Targets drawn from common words.
Moderate/Advanced: Increasing proportion of rare words.
Expert: Targets drawn from rare words mixed with distractors from the non-target topic.

3. Reinforcement Learning with Verifiable Rewards (RLVR)
Instead of human feedback, the models are trained using RLVR. A frozen "judge" policy (Google Gemini Flash-2.5) evaluates the model's output.

Reward Function: The reward is calculated based on the accuracy of the guess generation loop. It rewards correct target hits and penalizes non-target hits or guesses outside the board. The reward is bounded in $[-2, 1]$ .
Algorithm: The authors employ the DAPO (Deepseek-Adaptive Policy Optimization) objective with the GRPO advantage estimator. They fine-tune three Qwen3 models (1.7B, 4B, and 8B) using full-parameter fine-tuning over 2 epochs (230 update steps). No KL-divergence regularization is used.

Key Contributions

Creativity Training via Verifiable Rewards: The paper demonstrates that creativity can be trained effectively using a structured game environment with objective rewards, eliminating the need for human preference annotation.
Scale-Dependent Precision-Diversity Trade-off: The study reveals a critical finding: the effect of associative training on model capabilities is dependent on model scale.
- Smaller Models (1.7B, 4B): Gain substantial improvements in reasoning precision (math and science tasks) but often experience a collapse in creative diversity (vocabulary size and semantic diversity).
- Larger Model (8B): Gains consistent improvements in creativity metrics (divergent thinking, narrative generation) with only minor degradation in reasoning precision.
Scalable Solution: The approach provides a practical lever for strengthening LLMs across different sizes, delivering reasoning gains for smaller models and creativity gains for larger ones using a single, unified training recipe.

Results

The models were evaluated on 10 creativity benchmarks (covering divergent/convergent thinking, humor, metaphor, and creative writing) and 4 reasoning benchmarks (arithmetic, competition math, graduate science).

Creativity Gains: The 8B model showed modest but consistent gains across 8 of 10 creativity benchmarks. It was the only model to improve in divergent diversity (DAT DSI and vocabulary size). However, it regressed on NYT Connections, attributed to an "overthinking" tendency where the model violated task constraints due to extended reasoning traces.
Reasoning Gains: The smaller models (1.7B and 4B) showed substantial improvements on reasoning tasks. The 1.7B model improved on all four reasoning benchmarks (e.g., AIME24 accuracy rose from 0.43 to 0.57). The 4B model improved on three. Conversely, the 8B model saw declines in reasoning performance (e.g., AIME24 dropped from 0.80 to 0.67).
Mechanism: The authors attribute this split to the interaction between associative learning and RLVR. Smaller models, which previously struggled with hard problems, learn to solve the "easier" problems within reach with high precision. Larger models, already capable of solving these, are pushed by the reward signal to explore diverse, harder associations, thereby enhancing creativity at the cost of strict precision.

Significance and Claims

The paper claims to present a scalable and effective solution for training LLMs for creativity without human judgment. By casting creativity as a verifiable-reward game, the authors demonstrate that a single training paradigm can yield distinct, beneficial outcomes depending on the model's scale:

For smaller models, the method acts as a precision engine, sharpening reasoning capabilities.
For larger models, the method acts as a diversity engine, broadening creative associations.

The authors conclude that this scale-dependent trade-off is a key property of associative RLVR training. They emphasize that their approach lowers the resource barrier for creativity research by relying on synthetic data and verifiable rewards rather than costly human annotation, making it accessible for models as small as 1.7B parameters. The study cautions that while creative diversity improves, precision may trade off, suggesting these models should not be used in contexts requiring strict factual accuracy (e.g., medical or legal advice).