💻 computer science

The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost

This study demonstrates that while strategic model selection and increased reasoning effort significantly enhance automated scoring accuracy for high school mathematics conversations, ensembling via self-consistency yields no substantial gains, with specific low-cost models offering the optimal balance between performance and expense.

Original authors: Scott Frohn

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Scott Frohn

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading hundreds of short math conversations where students explain their thinking. You want to use an AI robot to do this grading for you, but you have two big worries: Will the robot be accurate? and Will it cost too much money?

This paper is like a lab experiment where researchers tested different "settings" on various AI robots to see which ones were the best at grading these math conversations. They compared robots from two big companies (OpenAI and Google) and tested two main strategies:

The "Committee" Strategy (Self-Consistency): Asking the same robot to grade the same answer multiple times and taking a vote.
The "Deep Thinker" Strategy (Reasoning Effort): Telling the robot to "think harder" before giving a grade.

Here is what they found, explained simply:

1. The "Committee" Strategy Didn't Help Much

The Idea: If one robot is a bit confused, maybe asking it to grade the same answer five times and taking the majority vote will smooth out the mistakes. It's like asking five friends to guess the answer to a riddle; if they all agree, you feel more confident.

The Reality: The researchers found that these AI robots are actually very stubborn. Once they decide on an answer, they stick to it, even if they are wrong.

The Analogy: Imagine a robot that is convinced the sky is green. If you ask it 10 times, it will say "green" 10 times. Asking it more times doesn't make it realize the sky is actually blue; it just makes you pay for 10 answers instead of one.
The Result: Asking the robot to vote on itself (from 1 time up to 7 times) did not improve accuracy. It just increased the cost. The only thing that helped slightly was making the robot a little more "random" (changing a setting called temperature), but simply asking it to vote more didn't fix the errors.

2. The "Deep Thinker" Strategy Was Hit or Miss

The Idea: Newer AI models can be told to "think step-by-step" before answering, similar to how a human might scratch out a math problem on paper before writing the final answer.

The Reality: This worked, but it depended entirely on which robot you used.

The "Over-Thinkers": For some cheaper, smaller robots, telling them to "think harder" actually made them worse at grading. They started over-analyzing simple answers and getting confused.
The "Smartest" Robot: The most powerful robot (Gemini 3.1 Pro Preview) was already so good that telling it to think harder didn't really change its score. It was accurate whether it thought for a second or a minute.
The General Trend: Overall, letting the models think a bit more did help accuracy, but the improvement wasn't a straight line. Sometimes thinking more helped, sometimes it didn't.

3. The "Price vs. Performance" Sweet Spot

The researchers drew a map (called an "efficiency frontier") to show the best balance between how good the grade was and how much it cost.

The Cheap Winners: The smallest, cheapest robots (GPT-5.4 Nano and Mini) with zero "deep thinking" turned out to be the best value. They were surprisingly accurate and cost pennies per grading task.
The Expensive Champion: The most powerful robot (Gemini 3.1 Pro Preview) gave the absolute highest accuracy, but it cost about 4 to 15 times more than the cheap robots.
The Middle Ground: Telling the cheaper robots to "think harder" usually just made them more expensive without making them much more accurate.

The Bottom Line

If you want to grade student math conversations automatically:

Don't ask the AI to vote on its own answers multiple times; it wastes money without fixing mistakes.
Do pick the right robot for your budget. If you need the absolute best accuracy and money is no object, use the most powerful model. But if you need to grade thousands of papers cheaply, a smaller, "lazy" (non-thinking) robot is often just as good and much cheaper.

Important Note: This study only looked at high school math conversations with simple "yes/no" checklists. The results might be different for essays, history questions, or more complex grading scales. Also, the prices mentioned are based on current API rates, which can change.

1. Problem Statement

Automated scoring of constructed-response questions (e.g., short answers, conversations) is critical for scalable educational assessment but faces challenges regarding consistency, cost, and accuracy. While Large Language Models (LLMs) have shown promise in zero-shot or few-shot scoring, two specific optimization strategies remain under-explored in educational contexts:

Self-Consistency (Ensembling): The hypothesis that aggregating multiple model outputs (majority voting) improves accuracy by smoothing out noise.
Reasoning Effort: The use of "thinking" models that generate internal Chain-of-Thought (CoT) traces before answering, with the assumption that deeper deliberation yields higher accuracy.

Prior research suggests these methods work well for objective reasoning tasks (like math problems with single correct answers) but may not translate to rubric-based scoring, where correctness involves subjective judgment. Furthermore, the cost-benefit trade-off of these strategies in large-scale educational deployments is unclear.

2. Methodology

Data and Context

Dataset: 900 student conversations from Khan Academy's "Explain Your Thinking" (EYT) items in high school mathematics (Algebra I and Geometry).
Structure: Each item involves a selected-response question followed by an AI-guided conversation.
Scoring Rubric: Dichotomous (binary) checklists with 2–3 criteria per conversation.
Ground Truth: Scores generated by three trained human raters, with the majority vote serving as the ground truth (2,100 total criterion scores).

Models Evaluated

The study compared frontier and low-cost models from two providers (OpenAI and Google):

OpenAI: GPT-5.4 (Frontier), GPT-5.4 Mini, GPT-5.4 Nano.
Google: Gemini 3.1 Pro Preview (Frontier), Gemini 3 Flash Preview.

Experimental Design

The study manipulated two variables:

Self-Consistency (Ensemble Size, $j$ ): Tested ensemble sizes of $j=1, 3, 5, 7$ $j = 1, 3, 5, 7$ .
- Constraint: Temperature sampling ($temp=1$) was used for ensembling to induce variance. Deterministic calls ($temp=0$) served as a baseline.
- Note: Reasoning models (OpenAI's reasoning_effort and Google's thinking_level) were restricted to "none" or "minimal" for the self-consistency analysis to avoid prohibitive costs.
Reasoning Effort: Tested levels ranging from "none/minimal" to "low," "medium," and "high."
- Constraint: Analyzed as single calls ( $j=1$ ) at $temp=1$.

Metrics

Accuracy: Measured using Cohen's $\kappa$ (agreement with human ground truth) and Generalized Linear Mixed Models (GLMMs) for statistical significance.
Cost: Estimated based on API pricing (March 2026 rates), calculating input/output tokens and reasoning tokens per 1,000 calls.
Efficiency Frontier: A Pareto-optimal analysis plotting Accuracy ( $\kappa$ ) vs. Cost to identify optimal configurations.

3. Key Results

R1: Self-Consistency (Ensembling)

High Intra-Model Consistency: Models were highly consistent with themselves (Fleiss' $\kappa$ between 0.88 and 0.97) even at $temp=1$. This indicates that models produce stable scoring patterns regardless of randomness, rather than variable reasoning paths.
No Accuracy Gains: Increasing ensemble size from $j=1$ to $j=7$ produced no significant improvement in scoring accuracy ( $p = .379$ ).
Temperature Effect: Stochastic calls ($temp=1$) were significantly more accurate than deterministic calls ($temp=0$) for specific models (GPT-5.4 Mini/Nano), but ensembling did not add value beyond a single stochastic call.

R2: Reasoning Effort

Positive Linear Trend: Overall, higher reasoning effort correlated with higher accuracy ( $b=0.145, p < .001$ ).
Model Heterogeneity: The benefit varied significantly by model family:
- Gemini 3.1 Pro Preview: Maintained the highest accuracy ( $\kappa \approx 0.794$ ) across all reasoning levels, showing little sensitivity to the parameter.
- GPT-5.4 Nano/Mini: Performed better with no reasoning than with low reasoning. Accuracy dipped at "low" effort before recovering at "medium/high."
- Gemini 3 Flash: Showed a clear trend of improvement from minimal to medium reasoning, then plateaued.
Conclusion: Higher reasoning effort is not a universal fix; it can degrade performance in smaller models or specific configurations.

R3: Cost-Efficiency Analysis

Cost Variance: Costs varied widely based on model size and reasoning tokens.
- Lowest Cost: GPT-5.4 Nano (No reasoning) at $0.27 per 1,000 calls.
- Highest Cost: GPT-5.4 (High reasoning) at $6.12 per 1,000 calls.
Anomalies: Gemini 3 Flash Preview showed a counterintuitive cost spike at "medium" reasoning due to excessive thinking token generation (716 tokens) compared to "high" reasoning (417 tokens).
Efficiency Frontier:
- Best Value: GPT-5.4 Nano and Mini (No reasoning) offered near-frontier accuracy ( $\kappa \approx 0.75$ ) at a fraction of the cost.
- Best Accuracy: Gemini 3.1 Pro Preview (Low reasoning) offered the highest accuracy ( $\kappa \approx 0.794$ ) but at 4–15x the cost of the most efficient options.
- Inefficiency: Increasing reasoning effort for GPT-5.4 and mid-tier models generally added cost without commensurate accuracy gains, pushing them off the Pareto frontier.

4. Key Contributions

Debunking Self-Consistency for Scoring: The study provides empirical evidence that self-consistency (majority voting) does not improve automated scoring accuracy for rubric-based tasks, as models exhibit high internal consistency even when incorrect.
Reasoning Effort Nuance: It demonstrates that "more reasoning" is not always better. For some models (e.g., GPT-5.4 Mini/Nano), disabling reasoning features yields superior results compared to low-effort reasoning modes.
Cost-Accuracy Trade-off Mapping: The paper establishes a clear efficiency frontier, identifying that low-cost models without reasoning often provide the best balance for large-scale deployment, while high-cost frontier models are only justified for specific high-stakes accuracy requirements.
Practical Deployment Guidelines: It offers actionable advice for educational technologists: prioritize temperature sampling ($temp=1$) over ensembling, and carefully select reasoning levels based on the specific model family rather than assuming defaults are optimal.

5. Significance and Limitations

Significance: The findings challenge the prevailing assumption that "ensemble methods" and "deep reasoning" are universally superior for LLM applications in education. It shifts the focus toward strategic model selection and parameter tuning as the primary levers for optimizing cost and accuracy.
Limitations:
- Scope: Limited to dichotomous scoring in high school math; results may not generalize to polytomous essay scoring or other subjects.
- Temporal Validity: Cost and performance figures are specific to the model versions and API pricing available in March 2026; rapid model iteration may alter these figures.
- Ensembling Scope: Only intra-model ensembling was tested; inter-model ensembling (combining different architectures) was not explored.

Conclusion: For automated short-answer grading, the optimal strategy is often a single call using a low-cost model (e.g., GPT-5.4 Nano) with temperature sampling and no/low reasoning effort, rather than expensive ensembling or high-effort reasoning configurations.