💻 computer science

HotComment: A Benchmark for Evaluating Popularity of Online Comments

This paper introduces HotComment, a multimodal benchmark that evaluates online comment popularity by integrating content quality, popularity prediction, and user behavior simulation, alongside the proposed StyleCmt framework to model socially resonant stylistic alignment.

Original authors: Yafeng Wu, Yunyao Zhang, Liliang Ye, Guiyi Zeng, Junqing Yu, Chen Xu, Zikai Song

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Yafeng Wu, Yunyao Zhang, Liliang Ye, Guiyi Zeng, Junqing Yu, Chen Xu, Zikai Song

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a massive, noisy party where thousands of people are watching a video and shouting comments. Some comments get thousands of cheers (likes), while others get ignored. The big question is: Why do some comments go viral while others flop?

This paper, titled "HotComment," argues that the current way computers try to write these comments is like a robot trying to guess what's funny without ever having been to a party. The authors say existing tools only check if the grammar is correct or if the words sound similar to other comments. They miss the "vibe."

Here is a breakdown of their solution using simple analogies:

1. The Problem: The "One-Size-Fits-All" Mistake

Imagine you tell a joke to a group of engineers, and then you tell the exact same joke to a group of comedians. The engineers might laugh politely, but the comedians might roar with laughter.

The Paper's Point: Popularity isn't just about the words you say; it's about who hears them, where they are, and how they react.
The Flaw: Current AI judges just look at the text. They don't understand that a comment that works on a news site might fail on a video game site.

2. The Solution: The "HotComment" Benchmark

The authors built a new testing ground called HotComment. Think of this as a simulated social media universe with three specific ways to grade a comment, rather than just one.

Dimension 1: Content Quality (The "Talent" Test)
Instead of just checking if the sentence makes sense, they check four specific "flavors":
- Linguistic Expression: Is it witty, rhythmic, or poetic?
- Creative Imagination: Does it connect two weird ideas in a new way?
- Emotional Resonance: Does it make you feel something (anger, joy, empathy)?
- Social/Cultural Influence: Does it use memes or references that people in that specific community understand?
- Analogy: It's like judging a chef not just on whether the food is cooked, but on the seasoning, the plating, the story behind the dish, and whether it fits the restaurant's theme.
Dimension 2: Popularity Prediction (The "Crystal Ball")
They trained a computer model on millions of real-world interactions. This model acts like a weather forecaster. It looks at the comment and the video, then predicts: "Based on how people usually behave on this specific platform, how many 'likes' will this get?"
Dimension 3: User Behavior Simulation (The "Crowd of Avatars")
This is the most unique part. Instead of one AI judge, they created a virtual crowd of thousands of AI "people" (agents).
- These agents have different personalities, jobs, and backgrounds (e.g., a student from New York, a retiree from a rural area).
- They simulate how a real, diverse audience would react. If the comment appeals to the "students" but bores the "retirees," the simulation catches that nuance.
- Analogy: It's like testing a new song in a focus group of 1,000 different people before releasing it to the world, rather than just asking one music critic.

3. The New Tool: "StyleCmt"

To help AI write better comments, the authors invented a framework called StyleCmt. They got the idea from physics, specifically how waves interact.

The Wave Analogy: Imagine the comments on a video create a "sound field" or a "vibe."
- Some comments are like constructive waves: They line up perfectly with the crowd's mood, making the sound (popularity) louder.
- Some are destructive waves: They clash with the mood and cancel each other out.
How StyleCmt Works: It analyzes the "waves" of the top comments already on the video. It then plans the new comment so its "waves" (style, humor, emotion) interfere constructively with the crowd's vibe. It doesn't just write a sentence; it tunes the sentence to match the frequency of the audience.

4. The Results

When they tested this new method against standard AI models:

The Standard AI: Wrote grammatically correct comments that felt robotic or missed the cultural context.
The StyleCmt AI: Wrote comments that felt like they were written by a human who "got the room."
The Score: The new method significantly improved scores in all categories. It made the AI better at being funny, more creative, more emotional, and culturally relevant. It also predicted that these comments would get more engagement (likes) in the simulation.

Summary

The paper says: To make AI write popular comments, you can't just teach it grammar. You have to teach it to understand the "vibe" of the room, simulate how a diverse crowd will react, and tune its writing like a radio to match the frequency of the audience. They built a new test (HotComment) to prove this works and a new tool (StyleCmt) to make it happen.

1. Problem Statement

Evaluating the popularity of online comments is a complex challenge that existing benchmarks fail to address adequately. Current evaluation methods primarily rely on:

Lexical/Semantic Similarity: Metrics like BLEU or BERTScore that measure overlap with human references but ignore stylistic nuance.
Isolated Stylistic Metrics: Assessing humor or creativity in isolation without considering how these elements interact with specific audience demographics and platform contexts.

The Core Gap: Popularity is not an intrinsic property of text; it is a conditional outcome ( $\pi(c|x, p, a)$ ) dependent on the comment ( $c$ ), source content ( $x$ ), platform context ( $p$ ), and audience state ( $a$ ). Existing frameworks lack the ability to model heterogeneous user preferences, platform-specific engagement dynamics, and the constructive/destructive interference of multiple stylistic elements simultaneously.

2. Methodology

The paper introduces two main components: the HotComment Benchmark and the StyleCmt Framework.

A. HotComment Benchmark

HotComment is a large-scale, multimodal dataset and evaluation framework designed to quantify comment popularity through three complementary dimensions:

Dataset Construction:
- Sources: Collected from NetEase News, Tencent News, and Bilibili.
- Scale: ~43,000 articles, 34,000 videos, and 1.4 million content-comment pairs.
- Labeling: "Popular" comments are defined as top-15 by likes (exceeding 10% of the top comment or 2,000 likes). "Non-popular" comments are low-engagement posts from the same day.
Three-Dimensional Evaluation Framework:
- Content Quality: Evaluates semantic similarity to ground-truth popular comments and assesses four interpretable stylistic dimensions:
  - Linguistic Expression (LE): Humor, irony, metaphor, rhythm.
  - Creative Imagination (CI): Originality, associative thinking, surrealism.
  - Emotional Resonance (ER): Empathy, affective tone, intensity.
  - Social & Cultural Influence (SCI): Memes, cultural references, intertextuality.
- Popularity Prediction: Uses models trained on real-world interaction data (fine-tuned BERT with MLP heads) to predict the likelihood of a comment becoming popular on specific platforms, capturing platform-level trends.
- User Behavior Simulation (UBS): An agent-based framework that simulates heterogeneous audience responses. It models the audience composition (e.g., interested vs. casual users) based on real-world demographic statistics and regional internet data to estimate engagement scores (thumbs-up/votes) rather than relying on a generic LLM judge.

B. StyleCmt Framework

To address the generation challenge, the authors propose StyleCmt, a framework inspired by the Wave Interference Model and Uses and Gratifications Theory. It treats stylistic elements as waves in a vector space that undergo constructive or destructive interference.

Resonance Field Construction: Historical comments are decomposed into the four stylistic dimensions (LE, CI, ER, SCI) to create a "Resonance Field" ( $\Psi_0$ ), representing the community's baseline preference (intensity and orientation vectors).
Interference-Driven Planning: A candidate comment's stylistic vectors are aligned with the Resonance Field. The framework calculates interaction scores (dot products) to identify combinations that maximize constructive interference (alignment) and minimize destructive interference.
Coherent Superposition: The generator produces multiple candidates by sampling from the aggregated stylistic space, ensuring the output aligns with the community's dominant tendencies.
Collapse and Emission: The candidate with the highest coherence score (cosine similarity to the baseline preference) is selected and refined for the final output.

3. Key Contributions

HotComment Benchmark: A comprehensive, multimodal dataset with a novel three-dimensional evaluation framework (Content Quality, Popularity Prediction, User Behavior Simulation) that moves beyond simple text overlap to model real-world engagement dynamics.
StyleCmt Framework: A novel generation method that models stylistic interactions as wave interference, enabling models to synthesize comments that resonate with collective audience preferences by amplifying compatible stylistic dimensions.
Empirical Validation: Extensive experiments demonstrating that StyleCmt significantly improves the realism and engagement alignment of generated comments across various Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs).

4. Experimental Results

The authors evaluated various models (Qwen, LLaMA, Mistral, ChatGPT-4o) on the HotComment benchmark.

Performance Gains: StyleCmt consistently outperformed baselines (including Chain-of-Thought and Few-Shot prompting) across all metrics.
- Content Quality: Significant improvements in BLEU-1, METEOR, and BERTScore F1.
- Stylistic Resonance Score (SRS): Substantial gains in all four dimensions (LE, CI, ER, SCI), with the most notable improvements in Creative Imagination and Social/Cultural Influence.
- Engagement Metrics: Large increases in Popularity Prediction scores and User Behavior Simulation (UBS) scores. For example, on Qwen2.5-14B, UBS improved by 20.39% and SRS by 17.38%.
Multimodal Capability: The framework proved effective for video-comment generation (MLLMs), where open-source models enhanced with StyleCmt achieved performance comparable to or exceeding closed-source baselines in engagement metrics.
Comparison: Unlike CoT or Few-Shot prompting, which offered marginal or inconsistent gains, StyleCmt provided stable, model-agnostic improvements in both semantic quality and social alignment.

5. Significance

Redefining Evaluation: HotComment shifts the paradigm from static text quality assessment to dynamic, context-aware popularity modeling, acknowledging that audience heterogeneity and platform context are critical to content success.
Bridging the Gap: It addresses the disconnect between "high-quality" text and "popular" text by explicitly modeling the social and stylistic mechanisms that drive engagement.
Generative AI Advancement: StyleCmt provides a principled approach for controllable text generation, demonstrating that modeling the interaction of stylistic features (rather than just the features themselves) is key to generating socially resonant content.
Future Research: The benchmark establishes a robust foundation for developing "popularity-aware" generation systems, crucial for applications in social media management, content recommendation, and human-AI interaction studies.