lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation

Technical Summary: SemEval-2026 Task 1 (MWAHAHA)

Problem Statement
The paper addresses the persistent difficulty of computational humor generation, which stems from two primary challenges: the inherent subjectivity of "funny" and the tendency of Large Language Models (LLMs) to memorize and repeat high-frequency templates rather than generate novel content. Humor is audience-dependent, varying significantly by culture, demographics, and context, leading to noisy supervision where annotator agreement is often low (e.g., Fleiss' $\kappa \approx 0.49$ for pun-related data). Furthermore, unconstrained generation often results in mode collapse, where models default to a small set of familiar jokes. The SemEval-2026 Task 1 (MWAHAHA) specifically targets constrained humor generation (e.g., conditioning on headlines or specific words) across English, Spanish, and Chinese, evaluating systems via human preference judgments in an arena-style setting. The authors argue that for constrained tasks, the bottleneck is not raw generation but the reliable selection of the optimal candidate for a target audience.

Methodology
The authors propose a two-stage "generate-many → select-best" pipeline, treating humor generation as a diversity maximization problem followed by a preference learning problem.

1. Candidate Generation:

   • Strategy: A multi-step prompting approach inspired by brainstorming techniques: generating associations, expanding them, and merging them into final jokes.
   • Diversity & Constraints: An ensemble of models (Claude-Sonnet-4, Claude-Ops-4.5, and GPT-5) is used with a temperature of 1.0. A "tail-focused sampling" trick draws candidates from lower-probability regions to increase novelty.
   • Filtering: 50 candidates are generated per instance. Hard constraints (keyword inclusion, length) are enforced deterministically. Near-duplicates are removed via embedding similarity, retaining only the centroid of each cluster.

2. Preference Modeling & Selection:

   • Interpretable "Humor Basis": Instead of relying solely on raw text embeddings, the authors construct a compact, interpretable feature basis (17 features in the final run). This is derived by prompting an LLM to generate "difference hints" between joke pairs, embedding these hints, clustering them (DP-means), and performing semantic de-duplication.
   • Feature Extraction: The basis is used to score jokes in two modes:
     • Pointwise: Scoring a single joke against the rules.
     • Pairwise: Comparing two jokes to determine which aligns better with specific rules.
   • Model Training: Lightweight preference predictors are trained using L1 (LASSO) for feature selection and L2-regularized regression for stability. The authors prioritize pairwise-based regression as the main approach, as it yields better global ranking stability despite higher computational costs ($\Theta(n^2)$ vs. $\Theta(n)$). They employ optimizations like prioritizing comparisons between disconnected components and dissimilar candidates to reduce the number of required evaluations to $\Theta(n \log n)$.
   • Global Ranking: Final rankings are aggregated using Bradley-Terry-Luce models and Elo-style procedures via the Evalica toolkit.

Key Contributions

• Preference Data Release: The release of $\sim$2.5K human pairwise judgments collected via the "Humor Arena" prototype, alongside existing Reddit-based and ScaleAI-based datasets.
• Preference-First Framing: A formalization of constrained humor generation as a ranking problem driven by repeated pairwise decisions, arguing this provides more stable learning signals than absolute scoring.
• Interpretable Pipeline: A method to convert labeled comparisons into a compact feature basis, enabling the training of lightweight, interpretable preference models.
• System Implementation: The "lmfaoooo" system, which integrates the generation and selection workflow, with all prompts, candidate pools, and artifacts publicly released.

Results

• Within-Dataset Performance: On three datasets (Reddit, ScaleAI, and Humor Arena), pairwise feature-based models consistently outperformed both pointwise baselines and "NoBasis" (direct LLM judge) baselines. For instance, on the Humor Arena dataset, the pairwise model achieved 77% accuracy compared to 45% for the pointwise model and 37% for the NoBasis model.
• Cross-Domain Transfer: Pairwise models demonstrated stronger transfer capabilities than pointwise baselines. While transfer was partial and asymmetric (indicating both universal and community-specific humor components), pairwise models maintained higher accuracy in off-diagonal test settings.
• SemEval-2026 Task 1 Performance: The system achieved 1st place in the English and Chinese subtasks and 2nd place in the Spanish subtask.
  • English ELO: 1041 (95% CI [1009, 1064])
  • Chinese ELO: 1081 (95% CI [1031, 1127])
  • Spanish ELO: 1091 (95% CI [1053, 1121])

Significance and Claims
The paper posits that the future of humor generation relies less on improving raw text generation capabilities—which modern LLMs already possess in abundance—and more on improving evaluation and preference modeling. The authors claim that their results support a "preference-first" view: generating diverse candidates is increasingly easy, but reliably identifying which candidate aligns with a specific audience remains the core difficulty. The use of an interpretable feature basis allows for targeted error analysis, revealing that while some humor features (e.g., clear punchlines) are universally selected, others are community-specific. The authors conclude that progress in this field will increasingly depend on robust preference modeling rather than solely on generation architecture improvements.