Technical Summary: Sustainability via LLM Right-sizing

Problem Statement

The rapid integration of Large Language Models (LLMs) into organizational workflows has generated significant concerns regarding energy consumption, financial costs, and data sovereignty. While existing AI benchmarks prioritize technical performance metrics (e.g., accuracy, generalization), they often fail to address the practical constraints of real-world deployment. Organizations lack empirical guidance on when a smaller, locally deployable model is "good enough" to perform occupational tasks effectively, affordably, and autonomously. This study addresses the gap between performance-maximizing benchmarks and the need for task- and context-aware sufficiency assessments that balance quality with sustainability (environmental, economic, and social).

Methodology

The authors developed an empirical evaluation framework to compare eleven LLMs (a mix of proprietary cloud-hosted and open-weight models) across ten representative occupational tasks.

1. Model Selection:
The study evaluated 11 models ranging from large proprietary systems (e.g., GPT-4o, Claude-3.7) to open-weight models capable of local deployment on single GPUs (e.g., Gemma-3, Phi-4, Llama-3.3).

2. Task Design:
Ten "one-shot" occupational tasks were designed to simulate typical knowledge work, categorized into:

• Aggregation: Summarizing texts, generating meeting minutes, extracting key information, creating comparison tables.
• Conceptual: Creating time schedules, drafting project proposals, writing to-do lists, planning presentations.
• Transformation: Writing formal/informal emails, rewriting for clarity and tone.
  Crucially, the study avoided prompt engineering or iterative refinement to reflect realistic "average user" interactions.

3. Evaluation Framework:
A dual-LLM architecture was employed:

• Execution Component: Each of the 11 models generated outputs for the 10 tasks.
• Evaluation Component: Three high-performing LLMs (GPT-4o, Gemini-2.0-Flash, o3-mini) scored the outputs blind to the source model.
• Criteria: Outputs were rated on a 1–10 Likert scale across 10 criteria grouped into three categories:
  • Core Qualities: Usefulness, Relevance, Completeness, Clarity, Tone Appropriateness.
  • Factual Integrity: Accuracy, Hallucination Freeness.
  • Ethical & Social Responsibility: Ethical Bias, Political Bias, Censorship.

4. Analytical Strategy:
The authors utilized one-way ANOVAs to test for significant differences in performance across models and task categories. Z-transformation was applied to normalize scores across tasks of varying difficulty. A K-means cluster analysis was performed to group models based on their performance profiles across the ten criteria. Sustainability metrics (cost per million tokens, deployment location, and compute-based CO2 proxies) were integrated to contextualize performance.

Key Results

1. Performance Hierarchy and Clustering:
The analysis identified three distinct model clusters:

• Cluster 1 (Premium All-Rounder): Contained only GPT-4o, which demonstrated consistently superior performance across all criteria, particularly in usefulness, clarity, and factual accuracy.
• Cluster 2 (Competent and Responsible): Included Claude-3.5/3.7, Gemini-2.0-Flash, Gemma-3, o3-mini, and Phi-4. These models showed solid overall performance with high scores in ethical criteria and tone, though slightly trailing GPT-4o in core utility metrics.
• Cluster 3 (Limited but Safe): Included DeepSeek-R1, Llama-3.3, Mistral-Nemo, and Qwen-2.5. While these models maintained high ethical safety scores, they frequently produced outputs requiring significant human revision due to issues with clarity, completeness, and usefulness.

2. Task-Type Effects:
Performance varied significantly by task category. Conceptual tasks (e.g., ideation, proposal drafting) yielded the lowest scores across most models, indicating a struggle with "generative reasoning." In contrast, Aggregation and Transformation tasks yielded higher performance, suggesting models are more robust in processing and restructuring existing information than in creating new conceptual frameworks.

3. Sustainability and Cost Trade-offs:

• Cost Disparity: Proprietary models (e.g., GPT-4.5-preview, Claude-3) incur costs ranging from $10 to $150 per million tokens, whereas open-weight models (e.g., Phi-4, Gemma-3) cost less than $1 per million tokens when hosted, with potential for near-zero marginal costs when run locally.
• Viability of Smaller Models: Models like Gemma-3 and Phi-4 achieved strong, reliable results on most tasks. The study posits that for many organizational contexts, these models offer a "good enough" balance of performance, cost-efficiency, and data sovereignty, making them viable alternatives to premium models.

4. Evaluation Consistency:
While three evaluators were used, statistical analysis (Intra-class Correlation Coefficient) indicated poor agreement between them. Consequently, the primary results were reported using GPT-4o as the sole evaluator, justified by its alignment with human judgment and the superior performance of other evaluators in the study.

Key Contributions

• Empirical Validation of "Sufficiency": The study provides evidence that smaller, locally deployable models can effectively perform a wide range of everyday occupational tasks, challenging the assumption that larger models are always necessary.
• Task-Aware Benchmarking: It demonstrates that model effectiveness is highly dependent on task type, arguing against one-size-fits-all benchmarking in favor of context-specific assessments.
• Sustainability Lens: The paper integrates cost, energy, and data sovereignty into the evaluation framework, offering a structured approach to "Green IS" (Green Information Systems) by quantifying the trade-offs between model size, performance, and operational sustainability.
• Scalable Evaluation Method: The authors propose a scalable, automated LLM-based evaluation framework that can be adapted for ongoing assessment of AI models in organizational settings.

Significance and Claims

The paper argues for a paradigm shift in AI deployment from "performance maximization" to "sufficiency assessment." The authors claim that:

1. Context Matters: The "best" model is not universally the largest; it is the one that aligns with specific task requirements, infrastructure constraints, and organizational priorities (e.g., privacy vs. peak performance).
2. Economic and Environmental Viability: Smaller, open-weight models offer a sustainable pathway for organizations, drastically reducing token costs and carbon footprints while maintaining data sovereignty.
3. Practical Guidance: The study offers actionable guidance for practitioners, suggesting that for many routine tasks, premium models are an over-provisioning of resources, whereas models like Gemma-3 and Phi-4 provide a robust, cost-effective alternative.

The authors remain modest, acknowledging limitations such as the reliance on a single LLM evaluator, the exclusion of direct energy measurements, and the focus on single-shot interactions. They conclude that responsible AI adoption requires asking not "which model is best?" but "which model is right?" for a specific context.