💬 NLP

Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks

This paper demonstrates that using bounding boxes to crop handwritten student responses significantly improves the grading accuracy and computational efficiency of Small Language Models on vision-based educational assessment tasks.

Original authors: Lachlan McGinness

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lachlan McGinness

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can read your handwriting, understand your thoughts, and even grade your homework. This is the exciting frontier of Artificial Intelligence in Education, where "Small Language Models" (SLMs) are the new heroes. Think of these models as clever, compact digital brains that can be run on a regular laptop or school server, keeping student data private and saving money compared to massive, cloud-based supercomputers. However, there's a catch: while these small brains are great at reading text, they sometimes get overwhelmed when looking at a whole page of messy, handwritten notes. It's like asking a detective to find a single specific clue in a room filled with thousands of unrelated objects; the sheer amount of visual "noise" can confuse them, making them miss the answer or waste a lot of energy trying to figure it out.

This paper by Lachlan McGinness from the Australian National University tackles exactly that problem. The author asks a simple but powerful question: What if we just showed the computer the specific part of the page where the answer is, instead of the whole messy page? To test this, the team used scanned exams from the 2025 Australian Physics Olympiad, where students had to write down a single word answer ("friction") to a question about walking. They tested eight different AI models, ranging from tiny 4-billion-parameter "puppies" to giant 72-billion-parameter "giants," and gave them two types of instructions: some were told to "think step-by-step" (a technique called Chain of Thought), while others were just told to give the answer. Crucially, they tested two visual setups: one where the AI saw the entire double-page exam scan, and another where the AI only saw a cropped "bounding box" around the specific question and answer area.

The results were a clear victory for the "crop" method. The study found that when the AI models were shown only the relevant bounding box, they became significantly more accurate at grading the answers, regardless of whether they were small or large models. In fact, showing the full page often confused the models, leading to lower scores. Interestingly, the "think step-by-step" instruction didn't help much; the models performed just as well (or sometimes better) when they were just told to give the answer directly. The most surprising win was efficiency: by cropping the image, the models needed to process far fewer "visual tokens" (the digital building blocks of images), which slashed the computational cost by more than half. For instance, the average number of tokens dropped from around 1,500 when viewing the full page to just under 700 when viewing the cropped box, reducing the energy required to grade each paper from 45 trillion operations down to about 17 trillion.

The paper explicitly argues against the idea that simply making the AI "think harder" (using Chain of Thought) is the best way to fix these errors; the data suggests that for this specific task, cleaning up the visual input is far more effective than changing the thinking process. They note that while the actual exam question was about human walking, the prompt text given to the models explicitly referred to the question as being about "snails" (likely due to a large snail diagram on the page), and the models were instructed to look for the answer in that context. They conclude that for schools wanting to use these affordable, private AI tools to grade handwritten exams, the secret sauce isn't necessarily a bigger model or a smarter prompt, but a simple pre-processing step: cropping the image to remove the clutter. This small change allows even the smallest, most energy-efficient models to perform like champions, making automated grading a much more realistic and practical tool for the future of education.

Technical Summary: Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks

Problem Statement
The integration of Small Language Models (SLMs) into educational settings offers distinct advantages regarding privacy, cost, and institutional control over data compared to large, cloud-hosted proprietary models. However, SLMs face significant challenges when applied to vision-based tasks, specifically the automated grading of handwritten student exams. Unlike digitized text, handwritten exams require the model to locate specific answers within a full page, interpret crossed-out text, and ignore irrelevant content from other questions.

The core technical bottleneck lies in how Vision-Language Models (VLMs) process images. VLMs tokenize images by partitioning them into patches, causing the number of visual tokens to scale linearly with image area. Consequently, processing a full A3 double-page exam scan consumes thousands of visual tokens. This high token count directly translates to increased computational cost (FLOPs) during inference, often dominating the forward pass. Furthermore, the visual density of a full page introduces distractions (e.g., other questions, diagrams, or scribbles) that can degrade the performance of smaller models.

Methodology
To investigate whether reducing visual search space improves SLM performance, the authors conducted an empirical study using 126 scanned handwritten responses to Question 1a of the 2025 Australian Physics Olympiad. The question required a simple short answer ("friction" or "friction force"), minimizing the need for complex reasoning and isolating the variable of visual extraction.

The study evaluated eight open-source VLMs ranging from 4B to 72B active parameters (including Llama 4 Scout, Mistral Small 3.2, Gemma 3 variants, and Qwen2.5-VL variants). The experiment utilized a 2x2 factorial design with two independent variables:

Prompting Strategy: Comparing standard prompting against Chain of Thought (CoT) prompting, where the model is instructed to "think step by step."
Input Modality: Comparing the presentation of the full A3 double-page scan versus a cropped bounding box containing only the relevant question and answer region.

The researchers manually defined bounding boxes based on the consistent layout of the exam papers, ensuring the crop captured student responses even with minor scanning variations. Performance was measured by grading accuracy against a human marking team and computational efficiency, calculated as total inference FLOPs based on input/output token counts using the approximation $C \approx 2nN$ .

Key Results
The experimental results, summarized across all models, yielded the following findings:

Impact of Bounding Boxes: The use of bounding boxes was the most significant factor in improving performance. Across all models, conditions utilizing bounding boxes achieved higher accuracy than those processing full-page scans. Specifically, the "No-CoT + Bounding Box" condition produced the highest accuracy (0.840 ± 0.020) on average.
Impact of Chain of Thought (CoT): Contrary to some expectations, CoT prompting did not provide a consistent benefit. For most models, the CoT conditions performed similarly to or slightly worse than non-CoT conditions. The only notable exception was Mistral Small 3.2, which showed improved performance with CoT, likely due to differences in its instruction-following training.
Computational Efficiency: Bounding boxes drastically reduced the computational cost. The average token count dropped from approximately 1,500 (full page) to roughly 680 (cropped). This reduction resulted in a decrease in average FLOPs from 45 trillion to 17 trillion per response.
Statistical Significance: The difference in accuracy between bounding box and non-bounding box conditions was statistically significant at the 99.8% confidence level.

Key Contributions and Claims
The paper claims that bounding boxes serve as a crucial pre-processing step for deploying SLMs in large-scale, vision-based educational assessments. The primary contributions are:

Empirical Validation: Demonstrating that reducing the visual search problem via cropping significantly improves grading accuracy for SLMs, likely by reducing visual noise and token overhead.
Efficiency Gains: Showing that this simple preprocessing step reduces computational costs by more than half without sacrificing (and often improving) accuracy.
Deployment Strategy: Arguing that for SLMs to be viable in education, architectural frameworks must compensate for their limitations by removing unnecessary visual clutter before the model processes the image.

Significance
The authors conclude that while SLMs are attractive for educational deployment due to their privacy and cost benefits, their effectiveness is currently limited by the computational and attentional demands of processing full-page documents. By implementing bounding boxes, institutions can optimize SLM deployment for learning analytics and automated grading, achieving a balance of high accuracy and low computational expense. The paper suggests that future research in this domain should prioritize techniques that minimize visual distractions to fully realize the potential of SLMs in education.

Technical Summary: Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks

More like this