DeliChess: A Multi-party Dialogue Dataset for Deliberation in Chess Puzzle Solving
The paper introduces DeliChess, a novel dataset of multi-party dialogues where participants collaboratively solve chess puzzles, demonstrating that group deliberation significantly improves collective accuracy while offering a rich testbed for studying complex reasoning and dialogue dynamics.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to solve a tricky chess puzzle. You sit down, think hard, and pick the move you believe is best. But what happens if you sit down with three friends, each of whom has their own idea of the best move, and you have to talk it through together to pick one final answer?
This is exactly what the researchers behind DeliChess wanted to study. They created a new "playground" (a dataset) to watch how groups of people think together when solving complex chess problems.
Here is a breakdown of their study using simple analogies:
1. The Experiment: A Chess "Group Hug"
Think of the chess puzzles as three different types of mental gym workouts:
- Tactical: Like a quick sprint. You need to spot a sudden trap or a quick win immediately.
- Positional: Like a long-distance run. You need to slowly improve your position and wait for the opponent to make a mistake.
- Endgame: Like a delicate dance. There are very few pieces left, and every tiny step matters.
The Process:
- Solo Warm-up: First, everyone solves the puzzles alone. This is their "base score."
- The Group Chat: Then, they jump into a group chat (like a Zoom call or a text thread) to argue, discuss, and change their minds.
- The Final Call: They submit one collective answer as a team.
The researchers recorded 107 of these group conversations, capturing every word typed, every move suggested, and how the group's final answer compared to their individual starting points.
2. The Big Discovery: "Two Heads (or Four) Are Better Than One"
The main finding is simple: Talking it through works.
Just like a group of hikers might find a hidden trail that a single hiker would miss, the groups in this study got significantly better at solving the puzzles after they discussed them.
- The "Tactical" Boost: The improvement was most dramatic on the "sprint" puzzles (Tactical). It seems that when things are fast and chaotic, having multiple people double-check each other's work is a superpower.
- The "Diversity" Factor: Groups where everyone started with very different ideas (some very good, some very bad) actually improved the most. It's like a band where everyone plays a different instrument; when they harmonize, the music is better than if everyone played the same note.
3. The Twist: Asking Questions Isn't a Magic Wand
The researchers were curious about a specific type of conversation: "Probing."
- What is probing? Imagine someone in the group saying, "Why did you pick that move?" or "What if we tried this instead?" These are questions designed to dig deeper.
Previous studies suggested that asking more questions always leads to better results. DeliChess found that this isn't always true.
- The Rollercoaster Effect: Groups that asked a lot of probing questions didn't necessarily get better answers. Instead, their results became more unpredictable.
- Sometimes, the questions helped them find a brilliant solution (a huge win).
- Other times, the questions confused them or led them down a rabbit hole, making their final answer worse than if they had just stayed quiet.
- The "Solution" Exception: There was one type of question that did help: questions specifically focused on the answer options (e.g., "Is move A better than move B?"). But general "why" questions didn't guarantee success.
4. Why This Matters
Before this study, most research on group discussions used simple tasks (like "Is this card red or blue?") or messy internet arguments where it's hard to tell who was right.
DeliChess is special because:
- It has a scoreboard: In chess, a computer (called an engine) can tell you exactly how good a move is. There is no arguing about whether a move was "good" or "bad"; the data is objective.
- It's complex: Chess requires deep strategy, not just simple logic.
The Takeaway
The paper tells us that while getting a group of people to talk through a problem usually leads to a better answer, how they talk matters. Just asking a bunch of questions doesn't automatically make a team smarter; it just makes the outcome more volatile. Sometimes the group hits the jackpot, and sometimes they crash.
The researchers released this data so other scientists can build better AI tools to help groups think together, using chess as a safe, clear, and measurable training ground.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.