💻 computer science

RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering

The paper introduces RECOM, a contamination-free dataset for evaluating open-ended Reddit question answering, to demonstrate that current automatic metrics face a fundamental validity-discrimination tradeoff where they can either distinguish genuine content from noise or rank model performance, but rarely both, due to inherent limitations in their representation design.

Original authors: Pushwitha Krishnappa, Amit Das, Vinija Jain, Aman Chadha, Tathagata Mukherjee

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pushwitha Krishnappa, Amit Das, Vinija Jain, Aman Chadha, Tathagata Mukherjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a talent show where five different robots are trying to answer open-ended questions like, "What's the best way to spend a rainy Sunday?" or "Why do cats act like they own the house?"

In the real world, there isn't one single "correct" answer. Instead, you have a crowd of humans (the Reddit community) who have already posted thousands of different opinions. Your job is to figure out which robot is doing the best job.

To do this, you use a "scorecard" (an automatic metric) to grade the robots. But this paper, RECOM, discovered a surprising problem: No single scorecard can do two jobs at once.

Here is the breakdown of the problem using simple analogies:

The Two Jobs of a Scorecard

The paper argues that a good evaluation tool needs to do two things:

The "Real vs. Fake" Test (Validity): Can the scorecard tell the difference between a robot's genuine answer and a random, nonsensical sentence?
The "Best Robot" Test (Discrimination): Can the scorecard tell you which of the five robots is actually the smartest?

The paper found that you can't have both. The tools that are great at spotting "fake" answers are terrible at ranking the robots. The tools that are good at ranking the robots are actually just measuring something trivial, like how long the robot talks.

The Two Types of Scorecards

1. The "Cosine Similarity" Scorecard (The Strict Gatekeeper)

How it works: This tool looks at the meaning of the words. It asks, "Does this answer sound like it belongs in the conversation?"
The Result: It is amazing at the "Real vs. Fake" test. It can easily tell the difference between a real human answer and random gibberish (like a coin toss).
The Flaw: It is terrible at ranking the robots. It gives all five robots almost the exact same score. It's like a teacher who gives every student an "A" because they all wrote something sensible, but you can't tell who wrote the best essay.

2. The "BERTScore" Scorecard (The Length Counter)

How it works: This tool looks at how well the robot's words overlap with human words.
The Result: It seems to rank the robots! One robot gets a 90, another gets an 85. It looks like it's doing a great job of picking a winner.
The Flaw: The paper discovered this ranking is a trick. The robot that got the "best" score was simply the one that wrote the shortest answers. Because the human answers on Reddit are often short, the robot that wrote short answers got a higher score just by matching the length, not necessarily the quality.
The Analogy: Imagine a contest where the judges give points for how many words you use. If the rule is "be concise," the robot that writes 10 words gets a perfect score, while the robot that writes 30 words gets a lower score, even if the 30-word answer is much wiser. When the researchers removed the "length" factor, the ranking disappeared, and all the robots looked the same again.

The "Noise Floor" Experiment

To prove this, the researchers created a "noise floor." They took the robots' answers and shuffled them randomly, pairing a robot's answer with a question it never saw.

They found that the "Real vs. Fake" gap (how much better a real answer is than a random one) was huge for some tools and tiny for others.
They found that the "Best Robot" gap (how much better one robot is than another) was often just an illusion caused by word count.

The "Human" Check

To make sure their computers weren't lying, they used three other AI models to act as human judges.

These "AI Judges" confirmed the same thing: They could easily tell the difference between a real answer and a random one (Validity).
But, just like the automatic scorecards, these AI Judges struggled to tell which of the five robots was actually the best (Discrimination). They gave them all very similar scores.

The Big Takeaway

The paper concludes that the problem isn't the robots; it's the rulers we are using to measure them.

The way these scorecards are built (their "representation design") makes them good at spotting nonsense but bad at spotting nuance.

If you want to know if an answer is real, use a tool like Cosine Similarity.
If you want to know which model is better, be very careful: the current tools might just be measuring how chatty the robot is, not how smart it is.

The authors suggest that in the future, we should report scores on both axes: "How well does this tell real from fake?" AND "How well does this rank the models?" so we don't get fooled by a scorecard that is just counting words.

Technical Summary: RECOM – A Validity–Discrimination Tradeoff in Automatic Metrics for Open-Ended Reddit Question Answering

Problem Statement

Automatic metrics are the standard for evaluating Large Language Model (LLM) outputs, yet they are implicitly tasked with two distinct functions: validity (distinguishing genuine content alignment from surface-level coincidence) and discriminative power (reliably ranking a better system against a worse one). In open-ended, opinion-driven tasks like conversational question answering, these two goals appear to be in tension. While metrics are routinely treated as a single "good" property, this paper posits that no single metric currently achieves both high validity and high discriminative power simultaneously. Existing evaluation datasets often rely on factoid questions with objectively verifiable answers (e.g., FreshQA, RealTimeQA), which fail to capture the nuance of community consensus in opinion-driven contexts. Furthermore, standard metrics often lack a "random-baseline floor," making it difficult to determine if a high score reflects genuine alignment or merely the statistical properties of fluent English text.

Methodology

To investigate this tradeoff, the authors introduce RECOM (Reddit Evaluation for Correspondence of Models), a contamination-free evaluation dataset and pipeline.

Dataset Construction: RECOM consists of 15,000 open-ended questions from r/AskReddit collected in September 2025. The collection window was chosen specifically to postdate the training cutoffs of all evaluated models (ranging from late 2023 to late 2024), ensuring no data contamination. For each question, the "ground truth" is defined not as a single correct answer, but as the set of authentic depth-1 community replies (direct human responses), preserving the full distribution of community perspectives.
Models Evaluated: Five open-source LLMs (7B–10B parameters) were evaluated: Mistral-7B, Qwen2.5-7B, Granite3.1-Dense-8B, Falcon3-10B, and Llama-3.1-8B. All models were prompted to generate answers under 50 words.
Evaluation Protocol:
- Multi-Reference Scoring: Each generated answer was scored against every depth-1 community reply for a given post.
- Metric Families: The study evaluated lexical (BLEU, ROUGE), semantic (BERTScore, MoverScore, Cosine Similarity), and inference-based (NLI) metrics.
- Random-Derangement Baseline: To measure validity, a random-derangement baseline was established where generated answers were paired with replies from different posts. This creates a "noise floor" to measure the separation between real alignment and random coincidence.
- Metrics of Interest: The authors calculated Cohen's $d$ for two axes:
  1. Validity: The effect size separating real generations from random pairings.
  2. Discriminative Power: The effect size separating the five different models from one another.
- LLM-as-Judge Validation: Three independent LLM judges (Phi-4, Gemma-3, Yi-1.5) from different developers were used to score the same data on "Relevancy" and "Faithfulness" to corroborate the validity axis.
- Length Control: To rule out response length as a confounding variable, the authors performed regression analyses and residualization of word counts.

Key Contributions

Identification of a Tradeoff: The paper identifies and quantifies a systematic validity–discrimination tradeoff. Metrics that excel at separating genuine alignment from noise (high validity) fail to rank models (low discriminative power), and vice versa.
The RECOM Dataset: A publicly available, contamination-free benchmark of 15,000 open-ended questions with authentic community multi-references, designed specifically to measure both metric axes against a random baseline.
Representation Design Hypothesis: The authors propose that the tradeoff stems from representation design. Specifically, contrastively-trained sentence encoders (used in cosine similarity) sharpen the distinction between related and unrelated text (validity) but collapse the subtle variations between different high-quality systems (discriminative power).
Reporting Framework: The authors advocate for reporting metrics on a two-dimensional space (validity vs. discriminative power) with an explicit random-baseline floor, rather than relying on raw scores.
Length Artifact Discovery: The study demonstrates that the apparent discriminative power of BERTScore precision is largely an artifact of response length. Once length is controlled, the ability of BERTScore to rank models collapses.

Results

Lexical Metrics (BLEU/ROUGE): These metrics showed high validity (clearly separating real from random, with Cohen's $d \approx 2$ ) but almost no discriminative power ( $|d| < 0.1$ ). Models clustered tightly, and rankings were unstable across metric variants.
Semantic Metrics:
- Cosine Similarity: Exhibited the highest validity (real-vs-random gap $\approx 24$ percentage points, $d \approx 2$ ) but failed to discriminate between models ( $|d| < 0.1$ ). The scores for all five models were nearly identical.
- BERTScore: Raw BERTScore precision appeared to have high discriminative power ( $|d|$ up to 0.63), ranking models differently. However, this ranking perfectly correlated with response length (shorter models scored higher). When response length was residualized, the discriminative power collapsed to $|d| = 0.09$ , rendering it negligible. Furthermore, raw BERTScore F1 scores (85–86%) were only 1–3 percentage points above the random baseline, indicating weak validity without rescaling.
- MoverScore: Showed moderate validity and moderate discriminative power that was not driven by length, but its overall signal was weaker than cosine similarity.
LLM-as-Judge: Three independent judges reproduced the validity gap (separating real from random with large effect sizes, $d \approx 2.0\text{--}3.5$ ) but, like the automatic metrics, failed to strongly separate the five models ( $|d|$ remained low).
Length Analysis: Response length explained very little of the real-vs-random gap (median $R^2 \approx 3\%$ ), confirming that the validity signal is not merely a verbosity artifact. However, length was the primary driver of the inter-model differences in BERTScore.

Significance and Claims

The paper argues that the validity–discrimination tradeoff is a property of the metrics, not the models. Because every metric scored the exact same model outputs, the inability to simultaneously validate alignment and rank systems is inherent to current metric designs.

The authors claim that the "good metric" ideal is unattainable with current single-metric approaches for open-ended tasks. They assert that:

Validity and discrimination are distinct questions: "Does this output engage with the content?" and "Which model is better?" are different inquiries that no single metric answers on open-ended tasks.
Baseline Rescaling is Critical: Raw scores (e.g., BERTScore F1 of 86%) often overstate alignment by 1–3 points over random chance. Rescaling against a random baseline reveals the true signal (6–13 points).
Length is a Confounder: Apparent model rankings based on BERTScore are largely artifacts of response length, not answer quality.

The paper concludes that evaluation practices must evolve to report metrics on both axes with explicit random baselines. It does not claim to solve the tradeoff but rather to expose it, suggesting that future work must address representation design to decouple these properties. The findings are limited to short-form, opinion-driven QA and may not generalize to factual QA or domains with objectively verifiable answers.