💬 NLP

TextReasoningBench: Does Reasoning Really Improve Text Classification in Large Language Models?

The paper introduces TextReasoningBench, a benchmark evaluating reasoning strategies for text classification across ten LLMs, which reveals that while simple reasoning methods offer marginal accuracy gains, complex strategies often fail to outperform baselines and incur prohibitive token costs, challenging the assumption that explicit reasoning universally benefits classification tasks.

Original authors: Xinyu Guo, Yazhou Zhang, Jing Qin

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinyu Guo, Yazhou Zhang, Jing Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart but expensive robot assistant (a Large Language Model, or LLM) that you want to hire to sort your mail. Some letters are simple, like "This is a bill" or "This is a vacation photo." Others are tricky, like a sarcastic joke or a subtle complaint hidden in a review.

For a long time, the tech world believed that to get the best results, you should always ask the robot to "think step-by-step" before answering. It's like asking a student to show their work on a math test, even if the answer is obvious. The assumption was: More thinking = Better answers.

This paper, TextReasoningBench, is like a rigorous science fair project where the researchers tested that assumption. They asked: "Does making the robot 'think harder' actually help it sort mail, or is it just wasting time and money?"

Here is the breakdown of their findings using some simple analogies:

1. The "Over-Thinker" Problem

The researchers tested seven different ways of making the robot think, ranging from a simple "Just answer" to complex methods like "Draw a tree of possibilities" or "Write a long essay before answering."

The Finding: For simple tasks (like sorting news articles by topic), asking the robot to think step-by-step often hurts its performance.
The Analogy: Imagine you are walking into a room to grab a glass of water. If you just walk in and grab it, it takes 2 seconds. But if you force yourself to "think step-by-step" (e.g., "First, I must lift my left foot. Then, I must check the floor texture. Then, I must calculate the angle of my arm..."), you might trip, spill the water, or forget why you went in the first place.
Result: On simple tasks, the "Over-Thinkers" (complex strategies like Tree-of-Thought) often got the answer wrong because they got distracted by their own internal monologue.

2. The "Price Tag" of Thinking

The researchers didn't just look at accuracy; they looked at cost. Every time the robot "thinks," it generates extra text (tokens), which costs money and time.

The Finding: Complex reasoning strategies often cost 10 to 100 times more than just asking for a direct answer, but the improvement in accuracy was tiny (sometimes just 1% or 2%).
The Analogy: It's like hiring a team of 100 lawyers to draft a contract that a single notary could have signed in 5 minutes. Sure, the 100 lawyers might find one tiny typo the notary missed, but you just spent a fortune to save a few cents.
Result: For most text classification tasks, the "Direct Approach" (Input-Output) is the most cost-effective. You get 95% of the benefit for 1% of the cost.

3. The "Small Brain" vs. "Big Brain"

The study compared small, cheaper models (like a smart calculator) with massive, powerful models (like a supercomputer).

The Finding:
- Small Models: When you force a small model to "think deeply," it often gets confused and makes worse mistakes. It's like asking a child to solve a complex physics problem by writing a novel; they just get lost in the words.
- Big Models: Massive models can benefit from thinking, but only on tricky, subjective tasks (like detecting sarcasm or irony).
The Analogy:
- Small Model: Asking a toddler to plan a wedding. They will likely forget the cake or invite the wrong people.
- Big Model: Asking a seasoned event planner. If the wedding is simple (just a cake), they don't need a plan. But if the wedding involves a surprise performance and a secret guest list (subjective nuance), their ability to "think through" the details actually helps.

4. The "Goldilocks" Zone of Reasoning

The researchers found that "more" isn't always "better." There is a sweet spot.

The Finding: A little bit of thinking (CoT) helps, but too much thinking (Long-CoT) leads to "overthinking," where the model starts hallucinating or contradicting itself.
The Analogy: It's like seasoning a soup. A pinch of salt (simple reasoning) makes it taste great. A whole shaker (complex reasoning) ruins it. And if you keep adding salt because you think "more is better," the soup becomes inedible.

5. The "Sarcasm" Exception

The one time where "thinking" really shines is when the task requires understanding human nuance, like sarcasm or sentiment.

The Finding: If a text says, "Oh, great, another traffic jam," a robot might think it's positive because of the word "great." But if the robot is forced to "think step-by-step," it might realize, "Wait, people don't love traffic jams. This is sarcasm."
The Analogy: For literal tasks (Is this a cat or a dog?), you don't need a detective. But for subtle tasks (Is this person being sarcastic?), you need a detective to read between the lines.

The Bottom Line

The paper challenges the popular idea that "thinking harder" is always the solution.

For simple jobs: Just ask the robot to do it. Don't make it write an essay first. It's faster, cheaper, and often more accurate.
For tricky jobs: A little bit of "thinking" helps, but don't overdo it.
The Takeaway: Don't use a sledgehammer to crack a nut. Sometimes, the simplest approach is the smartest one.

In short: Reasoning is a powerful tool, but like any tool, it's only useful when the job actually requires it. For most text classification tasks, the "lazy" (direct) approach is actually the most efficient.

1. Problem Statement

While explicit, step-by-step reasoning strategies (e.g., Chain-of-Thought, Tree-of-Thought) have become the dominant paradigm for enhancing Large Language Models (LLMs) on complex tasks like math and logic, their application to text classification remains underexplored.

The Assumption: There is a prevailing belief that deliberative reasoning (System-2 thinking) universally benefits all NLP tasks.
The Gap: Text classification often relies on semantic understanding and distributional patterns rather than explicit multi-step logical deduction. Furthermore, reasoning strategies incur substantial token and time costs.
The Core Question: Does explicit reasoning genuinely improve text classification performance, and is it computationally efficient compared to direct input-output (IO) prompting?

2. Methodology

The authors introduce TextReasoningBench, a systematic benchmark designed to evaluate the effectiveness and efficiency of reasoning strategies across diverse models and tasks.

2.1 Experimental Setup

Models: Evaluated 10 LLMs spanning a wide range of scales:
- Small Language Models (SLMs): GPT-4o-mini, Qwen-3-8B, Llama-3.1-8B, Gemma-3-4B.
- Big Models (BMs): GPT-5.2, Gemini-2.5-Flash, Qwen-3-Max, DeepSeek-V3.2, Kimi-K2.
Datasets: 5 Text Classification Datasets covering both objective and subjective tasks:
- Objective: AG News (topic classification), TREC-QC (question classification).
- Subjective: SST-2 (sentiment), SemEval 2018 (irony detection), iSarcasmEval (sarcasm detection).
Reasoning Strategies: Compared 7 distinct strategies in a zero-shot setting:
1. IO: Direct Input-Output (Baseline).
2. CoT: Chain-of-Thought (Linear reasoning).
3. SC-CoT: Self-Consistency CoT (Multiple chains + majority voting).
4. ToT: Tree-of-Thought (Tree-structured search with backtracking).
5. GoT: Graph-of-Thought (Graph-structured reasoning).
6. BoC: Bagging of Cues (Ensemble of independent reasoning fragments).
7. Long-CoT: Extended reasoning depth with self-reflection.

2.2 Evaluation Metrics

Beyond standard Accuracy and Macro-F1, the paper introduces two cost-aware metrics to quantify the trade-off between performance gain and token consumption:

Absolute Efficiency (AE): $F1 / \text{Total Tokens}$ . Measures overall cost-effectiveness.
Marginal Efficiency (ME): $\Delta F1 / \Delta \text{Tokens}$ $Δ F 1/Δ Tokens$ . Measures the incremental gain per additional token compared to the IO baseline.
- Positive ME: Reasoning adds value.
- Negative ME: Reasoning degrades performance or costs too much for the gain.

3. Key Contributions

TextReasoningBench: The first comprehensive benchmark evaluating 7 reasoning strategies across 10 LLMs and 5 text classification datasets.
Efficiency Metrics: Proposal of AE and ME to jointly assess performance and computational cost, moving beyond accuracy-only evaluations.
Empirical Evidence: Systematic demonstration that reasoning is not universally beneficial for text classification and is highly dependent on task type (objective vs. subjective) and model capacity.
Open Source: Release of code, prompts, and full evaluation results to facilitate reproducibility.

4. Key Results & Findings

4.1 Reasoning Does Not Universally Improve Performance

Objective Tasks: On tasks like AG News and TREC-QC, simple IO prompting is often competitive or superior. Complex strategies like ToT and GoT frequently degrade performance (e.g., ToT reduced GPT-4o-mini's performance on AG News by ~42% on SLMs) due to noise and error propagation.
Subjective Tasks: Reasoning shows more promise on subjective tasks (sentiment, sarcasm, irony) where implicit semantic inference is required. Strategies like SC-CoT and ToT often yield gains here, particularly for larger models.
Model Dependency: Small models (SLMs) are highly sensitive to complex reasoning, often suffering from instability. Large models (BMs) are more robust but still do not benefit uniformly from all strategies.

4.2 Reasoning is Often Highly Inefficient

Token Cost: Many strategies (SC-CoT, ToT) increase token consumption by 10× to 100×.
Efficiency Analysis:
- Absolute Efficiency (AE): The IO baseline dominates almost all settings. The massive token cost of reasoning is rarely offset by proportional performance gains.
- Marginal Efficiency (ME): While some specific cases show positive marginal returns (especially on subjective tasks with large models), the majority of configurations exhibit negative ME, meaning the extra cost yields diminishing or negative returns.

4.3 Longer Reasoning is Not Necessarily Better

Non-Monotonic Relationship: Performance does not scale linearly with reasoning length.
- Moderate Length: CoT and SC-CoT often provide the best balance.
- Excessive Length: Long-CoT and ToT often lead to "overthinking," where the model gets distracted by irrelevant details, leading to performance drops and high variance (instability).
Stability: ToT exhibits the highest variance (instability) across runs, while IO and SC-CoT are the most reproducible.

4.4 Few-Shot vs. Reasoning

Few-shot prompting often provides larger performance gains than reasoning strategies, particularly for smaller models. In some cases, few-shot IO outperforms complex reasoning chains.

5. Significance and Implications

Challenging the Paradigm: The paper challenges the assumption that "more reasoning is always better." It suggests that for text classification, the structural inductive bias of complex reasoning (e.g., tree searches) often misaligns with the cognitive demands of the task.
Cost-Aware Deployment: For practical applications, especially with limited compute budgets, IO prompting or simple CoT is often the most efficient choice. Complex strategies should be reserved for specific subjective tasks where implicit inference is critical and model capacity is sufficient.
Future Directions: The findings advocate for adaptive reasoning mechanisms that dynamically select the reasoning depth and structure based on task complexity and model size, rather than applying a "one-size-fits-all" reasoning approach.

In summary, TextReasoningBench reveals that while reasoning can be a powerful tool for specific subjective classification tasks, it is generally inefficient and often detrimental for objective text classification, particularly when applied to smaller models or via overly complex strategies.