💬 NLP

Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models

This study benchmarks humor alignment in five frontier Large Language Models by having them play Cards Against Humanity, revealing that while they outperform random chance, their modest agreement with human preferences and high inter-model consensus suggest their humor judgments are significantly influenced by structural artifacts rather than genuine human-like understanding.

Original authors: Yousra Fettach, Guillaume Bied, Hannu Toivonen, Tijl De Bie

Published 2026-04-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yousra Fettach, Guillaume Bied, Hannu Toivonen, Tijl De Bie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've invited five super-smart robots to your house for a game night. The game is Cards Against Humanity, a popular party game where players try to make each other laugh by matching a silly question (the "black card") with the funniest answer (the "white card") from a hand of ten options.

Usually, humans play this. But in this study, the researchers asked five of the world's most advanced AI models (like GPT, Claude, and Gemini) to play along with real humans. They wanted to see: Do these robots have a sense of humor like us? Or do they just think they're funny?

Here is what happened, broken down simply:

1. The Setup: A Digital Game Night

The researchers took nearly 10,000 rounds of games that real humans had played online. For every single round, they showed the same question and the same 10 answer choices to the five AI models.

The Human Player: Picked the card they thought was the funniest.
The AI Players: Each picked the card they thought was the funniest.

The goal was to see if the AI would pick the same card the human picked.

2. The Big Surprise: Robots Agree with Each Other More Than Us

You might expect the robots to try to mimic human laughter. But the results were weird:

The Robots vs. Humans: The AI models were only "right" about 13% to 18% of the time. If they were just guessing randomly, they'd get 10% right. So, they are slightly better than a random guess, but they are still mostly missing the human joke.
The Robots vs. Robots: Here's the kicker. The AI models agreed with each other way more often than they agreed with humans. Sometimes they agreed with each other nearly 60% of the time!

The Analogy: Imagine you and four friends are at a party. You all try to pick the funniest joke from a list.

You (the human) pick Joke A.
Your four robot friends all pick Joke B.
Even though you are the "expert" on what's funny, the robots are all nodding at each other saying, "Yes, Joke B is hilarious!" while you are confused.

3. Why Are They Doing This? (The "Robot Glitch")

The researchers dug into why the robots were picking these specific jokes. They found two main reasons the robots weren't acting like humans:

A. The "Position" Bias (The Seat at the Table)
Humans look at the meaning of the joke. The robots, however, seemed to have a favorite seat.

One robot always loved the card in Position 3.
Another always loved the card in Position 10 (the very last one).
It's like if a student always picked the third answer on a multiple-choice test, not because it was right, but because they liked the number three.

B. The "Topic" Bias (The Flavor of the Joke)
The robots also had a specific taste in humor that didn't match humans:

Humans: Liked jokes about politics, identity, and social issues.
Robots: Loved jokes about bodies, bodily fluids, and sex. They picked these types of jokes way more often than humans did.
Why? The researchers think the robots were "trained" to be polite and safe. So, when they try to be funny, they avoid the risky, edgy, political jokes that humans actually find hilarious in this game. Instead, they lean into "gross-out" humor because it feels "safer" to the AI's programming, even though it's not what humans actually prefer.

4. The Conclusion: The "Uncanny Valley" of Humor

The study concludes that while these AI models are amazing at writing code or summarizing text, humor is still a mystery to them.

They haven't learned to "get" jokes the way we do. Instead, they have developed their own "robot sense of humor" that is:

Consistent: They all agree with each other.
Flawed: They rely on simple tricks (like picking the 3rd card) and specific topics (gross-out humor) rather than understanding the complex social context of a joke.

The Takeaway:
If you want to know what a human thinks is funny, don't ask the AI. The AI is currently like a very smart alien who is trying to fit in at a human party but keeps making the same few jokes and sitting in the same chair, confusing everyone. They are getting closer, but they still have a long way to go to truly "get" the joke.

1. Problem Statement

Humor is a complex, culturally embedded dimension of human communication that relies on shared knowledge, timing, and the violation of expectations. While Large Language Models (LLMs) have achieved human-level performance in structured tasks (coding, writing), their ability to align with human humor remains largely unexplored.

The Core Question: Do frontier LLMs possess a "sense of humor" that aligns with human preferences, or do they converge on a distinct, human-misaligned profile driven by structural artifacts?
The Challenge: Humor is subjective and difficult to formalize. Existing benchmarks often focus on detection or ranking, but lack the interactive, preference-based context of real-world humor games.

2. Methodology

The authors propose a controlled, game-based evaluation framework using Cards Against Humanity (CAH) as a naturalistic testbed.

Experimental Setup

Dataset: The study utilizes the CAH Lab Gameplay Dataset, containing 9,894 rounds of gameplay collected between November 2023 and April 2025. Each round consists of a "black card" (prompt) and ten "white card" candidates.
Models Evaluated: Five frontier LLMs were tested:
- GPT-5.2
- Gemini 3 Flash
- Claude Opus 4.5
- Grok 4
- DeepSeek-V3.2
Procedure:
1. Each round was presented to the models as a structured selection task (select the funniest card from 10 options).
2. Replicates: To assess stability and position bias, each round was run twice ( $R=2$ ) with the order of the 10 white cards randomly permuted.
3. Filtering: Rounds with completion times <10s or >120s were excluded to ensure genuine human deliberation.
4. Demographics: Human player data was enriched with demographic attributes (gender, race, ideology, etc.) to analyze alignment across subgroups.

Metrics and Analysis

The study evaluates alignment along three axes:

Human–LLM Alignment: Measured by Accuracy Rate (ACC), calculating how often an LLM selects the same card as the human player.
LLM–LLM Alignment: Measured by Inter-model Pairwise Agreement (do different models agree with each other?) and Intra-model Consistency (does a model agree with itself across replicates?).
Behavioral Analysis:
- Position Bias: Chi-square tests to detect if models favor specific card positions (e.g., first or last).
- Topic Bias: Analysis of the semantic topics (e.g., bodily humor, politics, sex) of selected cards compared to human selections and the available pool.
- Predictive Modeling: Conditional logit models were fitted to determine if position and topic alone can predict model choices.

3. Key Contributions

Framework for Humor Alignment: Introduced a scalable framework where frontier models play the exact same CAH rounds as humans, allowing for direct comparison of humor preferences.
Quantification of Misalignment: Demonstrated that while LLMs outperform random baselines, their alignment with human humor is modest (13–18%).
Discovery of "LLM Humor": Revealed that models agree with each other significantly more often than they agree with humans, suggesting the emergence of a shared, but human-misaligned, humor profile.
Decomposition of Bias: Identified that position bias and content preferences (specifically for bodily/sexual themes) explain a substantial portion of inter-model agreement, questioning whether LLM humor is genuine or a structural artifact.

4. Key Results

Human–LLM Alignment

Low Accuracy: All five models achieved accuracy rates between 13% and 18%, significantly exceeding the random baseline (10%) but falling short of simple baselines like popularity-based selection (~19%).
No Demographic Heterogeneity: The low alignment was consistent across various socio-demographic groups (gender, race, political ideology), suggesting the misalignment is a general model property rather than a failure to match specific subgroups.
Ranking: Claude Opus 4.5 showed the highest alignment, followed by Grok and Gemini. DeepSeek and GPT trailed slightly.

LLM–LLM Agreement

High Inter-Model Consistency: Models agreed with each other in 21.4% to 44.9% of rounds.
High Intra-Model Consistency: Models were highly self-consistent across replicates (49.5% to 63.3%).
The Divergence: Crucially, LLMs agree with each other far more than they agree with humans. This indicates a convergence toward a distinct "LLM sense of humor."

Bias Analysis

Position Bias: All models exhibited significant non-uniform selection patterns.
- DeepSeek heavily favored Position 3.
- Grok favored the last position (10).
- Claude favored early positions.
Topic Bias:
- LLMs disproportionately selected cards related to bodily humor (31–40% vs. 21% for humans) and sexual themes (29–38% vs. 24% for humans).
- LLMs under-selected cards related to politics/society and identity/demographics compared to humans.
- GPT was an outlier, relying more on "miscellaneous" themes.
Predictive Power: Surrogate logit models using only position and topic flags could predict model choices with accuracies up to 36% (for Grok, DeepSeek, and Gemini), confirming that these biases drive a significant portion of the decision-making process.

5. Significance and Implications

The "Alignment Ceiling": The study suggests a current ceiling in human-LLM humor alignment. The gap between human and model preferences may stem from the safety-oriented fine-tuning (RLHF) of modern LLMs, which suppresses the transgressive, absurd, and boundary-pushing humor that defines games like CAH.
Structural Artifacts vs. Understanding: The high inter-model agreement and the success of simple bias-based predictors suggest that LLM "humor" may not be a deep understanding of social context but rather a reflection of structural artifacts (position bias) and training data distributions (preference for specific topics).
Future Directions: The authors argue that humor is a critical frontier for AI alignment. Future work should explore cross-cultural models, diverse humor formats (memes, satire), and whether specific human subgroups (defined by humor style rather than demographics) align better with LLMs.

Conclusion

"Cards Against LLMs" provides empirical evidence that while frontier LLMs can mimic human humor to a degree, they have converged on a distinct, structurally biased profile that diverges significantly from human preferences. This misalignment highlights the limitations of current RLHF procedures in capturing the nuance, transgression, and cultural specificity required for genuine humor generation and judgment.