💬 NLP

Comparing energy consumption and accuracy in text classification inference

This study systematically evaluates the trade-offs between accuracy and energy consumption in text classification inference, revealing that while LLMs often consume more energy than traditional models, they do not necessarily outperform them in zero-shot settings, and that inference runtime can serve as a practical proxy for energy usage.

Original authors: Johannes Zschache, Tilman Hartwig

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Johannes Zschache, Tilman Hartwig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy post office. Every day, thousands of letters (text messages) arrive, and you need to sort them into the correct bins (categories) so they get to the right department. You have two types of workers to help you:

The Local Experts (Traditional Models): These are workers who have been trained specifically on your post office's rules. They are fast, they use very little electricity, and they know exactly where to put the letters.
The Super-Geniuses (Large Language Models or LLMs): These are brilliant scholars who know everything about the world. They haven't been trained specifically on your post office, but they can guess the right bin just by reading the letter. However, they are slow, they need massive amounts of electricity to power their brains, and they sometimes overthink simple tasks.

This paper is a head-to-head race between these two types of workers to see who sorts the mail best while saving the most energy.

The Big Surprise: The Local Experts Often Win

The researchers set up a test using real letters from a German government office about radioactive waste. They asked both the Local Experts and the Super-Geniuses to sort 189 letters.

The Result: The Local Experts (specifically a simple model using "sentence embeddings") were the most accurate and used the least amount of energy.
The Shock: The Super-Geniuses (the massive AI models) used thousands of times more electricity than the Local Experts. In some cases, the Super-Geniuses were actually less accurate than the simple workers, even though they consumed enough energy to power a small house for a day.

The Analogy: It's like using a nuclear-powered submarine to deliver a single letter across the street. Sure, the submarine is incredibly advanced, but it's a terrible choice for the job. A bicycle (the traditional model) gets the job done faster, cheaper, and with zero carbon emissions.

The "Thinking" Trap

The study also looked at a new trend where AI models are told to "think" before they answer (like the DeepSeek models).

The Analogy: Imagine a worker who reads a letter, then spends 10 minutes writing a 5-page essay about why they think it goes in a specific bin, before finally putting it in the bin.
The Finding: This "thinking" process made the AI slower and used even more energy, but it didn't make the sorting more accurate. In fact, for simple text sorting, this extra thinking was just wasted effort.

The Hardware Factor: Bigger Isn't Always Better

The researchers tested these models on different computer "engines" (GPUs), from old, slow ones to brand-new, super-fast ones.

The Finding: Even with the fastest, most expensive engines, the Super-Geniuses still guzzled way more power.
The Rule of Thumb: The researchers discovered a simple trick: Time is Energy. If you know how long a task takes, you can pretty much guess how much electricity it used. If a model takes 10 seconds to sort a letter, it uses a certain amount of power. If it takes 100 seconds, it uses 10 times the power. You don't always need a fancy meter; just a stopwatch works.

What About Other Jobs?

The researchers worried, "Maybe this only works for radioactive waste letters." So, they tested the models on other tasks like sorting movie reviews, news articles, and emotions.

The Verdict: The pattern mostly held up. For simple tasks, the small, efficient models were usually better.
The Exception: When the task got very complex (like understanding deep human emotions or nuanced news), the Super-Geniuses did start to win on accuracy. But even then, you have to ask: Is the tiny improvement in accuracy worth the massive spike in electricity bills?

The Takeaway for the Real World

The authors argue that we are currently obsessed with making AI "smarter" (bigger models) without caring about the cost. They suggest a new way of thinking:

Start Small: Before you hire a Super-Genius, try a Local Expert. You might find they do the job just as well for a fraction of the cost.
Measure the Bill: Don't just look at how accurate the AI is; look at its energy "price tag."
Don't Overthink: If a simple answer works, don't force the AI to write a novel to get there.

In short: Just because you can use a giant, energy-hungry robot to sort your mail, doesn't mean you should. Sometimes, the best solution is a simple, efficient tool that gets the job done without burning down the planet.

1. Problem Statement

The rapid deployment of Large Language Models (LLMs) in Natural Language Processing (NLP) has raised significant concerns regarding energy efficiency and environmental sustainability. While prior research has extensively analyzed the energy costs of model training, the inference phase (the operational phase where models process real-world data) has received comparatively less attention.

The core problem addressed is the lack of systematic evaluation regarding the trade-offs between model accuracy and energy consumption during inference, particularly for text classification tasks. There is a prevailing assumption that LLMs are necessary for high-quality classification, but it is unclear if their significantly higher energy costs are justified by performance gains over traditional, lightweight machine learning models, especially in zero-shot settings.

2. Methodology

The study employs a systematic, empirical approach to compare various model architectures and hardware configurations.

Datasets:
- Primary Dataset (FKTG): A specialized dataset from German public administration regarding radioactive waste management (378 entries, 14 categories). This represents a real-world scenario with limited labeled data and domain-specific terminology.
- Secondary Datasets: To test generalizability, the authors replicated experiments on four standard benchmarks: AG News (news topics), Yelp (sentiment), Rotten Tomatoes (sentiment), and Emotion (emotion detection).
Models Evaluated:
- Traditional Models: Linear Logistic Regression and XGBoost (Gradient Boosting). These were trained on the data using three feature types: Bag-of-Words (BoW), TF-IDF (character-level n-grams), and pre-trained multilingual sentence embeddings (Sentence-BERT).
- Large Language Models (LLMs): A diverse set of transformer-based models (ranging from <1B to >70B parameters) including Llama 3.1, Qwen 2.5, Phi 3.5, Jamba, and DeepSeek distillation models.
- Inference Strategy: LLMs were evaluated in a zero-shot setting (no fine-tuning) to reflect scenarios where training is impractical. Traditional models were trained on the dataset to ensure a fair comparison of "out-of-the-box" LLMs vs. "task-adapted" traditional models.
Hardware & Infrastructure:
- Experiments were conducted on High-Performance Computing (HPC) clusters at TUD Dresden and Leipzig University.
- GPUs: NVIDIA H100 (Capella), A30 (Paula), and V100 (Clara).
- Deployment: Models were deployed using the vllm library. Experiments tested single-node vs. multi-node (pipeline parallelism) and varying GPU counts (1, 2, or 4 GPUs).
Measurement:
- Energy: Measured using the CodeCarbon package, tracking GPU power via NVML and estimating CPU power via TDP and utilization.
- Metrics: Accuracy, total energy consumption (Wh), and inference duration (seconds).
- Protocol: Each experiment was repeated 10 times with different random train-test splits to ensure statistical robustness.

3. Key Contributions

Empirical Evidence on Inference Efficiency: The study provides concrete data showing that for text classification, traditional models often outperform LLMs in energy efficiency while maintaining comparable or superior accuracy, particularly in specialized domains.
Hardware and Architecture Analysis: It quantifies the impact of GPU types (H100 vs. V100 vs. A30) and deployment strategies (single vs. multi-node) on energy consumption.
Runtime as a Proxy for Energy: The authors establish a strong linear correlation between inference runtime and energy consumption, suggesting that execution time can serve as a reliable proxy for energy usage when direct power measurement is unavailable.
Zero-Shot vs. Fine-Tuned Comparison: The paper highlights the asymmetry in current evaluations: LLMs are often compared in zero-shot mode against fine-tuned traditional models, yet LLMs still consume orders of magnitude more energy.
Open Source Framework: The authors released a reproducible codebase and measurement data to facilitate further research in sustainable AI.

4. Key Results

A. Accuracy vs. Energy Trade-off

Traditional Models Win on Efficiency: In the primary FKTG dataset, the best-performing model was a Linear model with Sentence Embeddings, achieving the highest accuracy (0.57) with moderate energy consumption.
LLM Inefficiency: Even the most energy-efficient LLMs consumed significantly more energy than traditional models. For example, the best small LLM (Qwen 2.5 7B) consumed 7x more energy than the most efficient traditional model (Linear TF-IDF) for a negligible accuracy gain.
Reasoning Models are Costly: DeepSeek models (which use "thinking" tokens for reasoning) exhibited the highest energy consumption (up to 702 Wh for Llama 70B) and longest runtimes, yet did not consistently achieve higher accuracy than non-reasoning LLMs.
Task Dependency: On general benchmarks (e.g., AG News, Emotion), LLMs sometimes outperformed traditional models in accuracy, but the energy cost was often prohibitive (e.g., 34 Wh vs. 0.002 Wh for a 0.05 accuracy gain).

B. Hardware and Deployment Impact

GPU Count: Increasing the number of GPUs reduced inference duration but did not reduce total energy consumption. In some cases, adding GPUs increased total energy due to the overhead of operating multiple chips.
Multi-Node Deployment: Splitting a model across two nodes (pipeline parallelism) increased energy consumption by a factor of 1.5x to 2.7x compared to single-node deployment due to communication overhead and sequential execution.
GPU Architecture: The efficiency gains of newer H100 GPUs over V100 or A30 were only evident for DeepSeek models that generate substantial text (reasoning). For single-token classification tasks, older GPUs (V100/A30) were often more energy-efficient.

C. Runtime-Energy Correlation

A linear regression analysis revealed an $R^2 > 0.98$ correlation between inference duration and energy consumption across all hardware configurations.
Formula: $Energy \approx (a + c \cdot GPUs) \cdot Duration$ .
Implication: Monitoring runtime is a practical, low-overhead method to estimate energy usage in production environments.

5. Significance and Implications

Sustainable AI Development: The study argues that "bigger is not better." For many NLP tasks, particularly text classification, lightweight traditional models offer a superior balance of performance and sustainability.
Policy and Deployment: Public administration and enterprise IT should adopt a "simple first" approach: start with traditional models and only deploy LLMs if they provide a measurable, significant accuracy improvement that justifies the exponential increase in energy costs.
Evaluation Standards: Model assessment must evolve to include inference efficiency as a primary metric alongside accuracy. The paper advocates for the integration of energy metrics into standard model selection workflows.
Future Research: The authors call for "Planet-Centered LLMs" that consider not just energy, but also water usage, rare-earth metals, transparency, and ethical data sourcing.

Conclusion: The paper demonstrates that high accuracy in text classification does not necessitate high energy expenditure. By systematically evaluating the trade-offs, the authors provide a framework for selecting models that align with both performance goals and sustainability objectives, challenging the default assumption that LLMs are the superior choice for all NLP tasks.