Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification

Imagine you are hiring a team of super-smart, chatty robots to sort a mountain of mail into "Good" and "Bad" piles. You don't just want them to sort the mail; you want them to tell you how sure they are about each piece. If a robot is only 50% sure, you might want to set that letter aside and have a human double-check it. This is the world of Large Language Models (LLMs) acting as classifiers. These AI systems are like brilliant but sometimes overconfident students; they can write essays and solve riddles, but when asked to categorize things, they need to be able to say, "I'm pretty confident," or "I'm guessing."

The big question scientists are asking is: How do we know if these robots are actually telling the truth about their confidence? If a robot says, "I am 95% sure," is that a real, calculated feeling, or is it just a lucky guess? To test this, researchers use a special scoreboard called the Accuracy-Rejection Curve. Imagine a sliding scale: as you tell the robot to be stricter (only accepting answers where it says it's 99% sure), the pile of accepted letters gets smaller, but the ones left in the pile are almost always correct. The goal is to find the "sweet spot" where you keep the most letters while still being super accurate. The problem is, measuring this scoreboard is tricky if the robot's answers are clumpy and repetitive.

---

The Robot's "95%" Habit

In this paper, the authors, Elena Merdjanovska and her team, discovered a weird quirk in how these AI robots express their confidence. When you ask a robot to say a number out loud (like "I am 95% confident"), it doesn't pick numbers randomly like a human rolling dice. Instead, it gets stuck in a rut.

Think of it like a student taking a multiple-choice test who, instead of thinking, just writes "95" on every single answer sheet because it feels like a safe, round number. The researchers found that a popular AI model (Qwen3-32B) only used eight unique numbers (0.6, 0.65, 0.7, 0.75, 0.85, 0.9, 0.95, and 0.98) to describe its confidence across thousands of questions. Even crazier? More than half of the time, it just said 95%.

This is a problem because it's like trying to tune a radio but the dial only clicks into eight specific spots. If you want to filter out the "bad" letters to get 98% accuracy, you might have to throw away 40% of your mail. But if the robot could give you a more precise number, like 96.3%, you might only need to throw away 28% of the mail. Because the robot is so "sparse" (stuck on so few numbers), it wastes a lot of good data.

The Trap of the "Smooth Line"

Here is where the plot thickens. The researchers found that the way scientists were measuring the robot's performance was actually lying to them.

To calculate the scoreboard (called AUARC), researchers usually draw a line connecting the dots on a graph. Most people use linear interpolation, which draws a smooth, straight line between the dots, assuming the robot's confidence changes gradually. But if the robot only has eight numbers, there are huge gaps between the dots! Drawing a smooth line through those gaps is like connecting the dots of a jagged mountain range with a straight ruler—it creates a fake, smooth hill that doesn't exist.

The authors showed that using this "smooth line" method made the robots look much better than they really were. In fact, when they switched to the correct method—stepwise interpolation, which draws a flat line until the next actual number appears—the rankings flipped completely. A method that looked like the "champion" with the smooth line dropped to last place with the stepwise line. It turns out, the smooth line was just an optical illusion that made the sparse robots look more flexible than they actually were.

The Magic "Logprobs" Trick

So, how do we fix a robot that only speaks in eight numbers? The team proposed a clever hack called Verbalization Logprobs.

Usually, when a robot says "95%," it just spits out the words "9" and "5." The standard way to use this is to just take the number 95. But the robot actually knows something else: it knows how likely it was to pick "9" versus "8," and how likely it was to pick "5" versus "6." These hidden probabilities are called logprobs.

The authors' new method is like asking the robot not just for the number, but for the feeling behind the number. Instead of just taking "95," they calculate a weighted average based on how sure the robot was about each digit. This turns the robot's clumpy, eight-number answer into a smooth, continuous stream of confidence.

The results were impressive. By using this trick, the robot's confidence scores became much more detailed (jumping from just 8 unique values to over 600 unique values!). This didn't cost any extra computer power or time; it just used information the robot already had. With this new method, the robot could find that "sweet spot" much better, improving its performance score by 2.3 points compared to the old way. It was like giving the robot a high-definition dial instead of a broken, clicky one, allowing it to sort the mail with much less waste.

The Takeaway

The paper concludes that we need to stop pretending these robots are smooth operators when they are actually quite rigid. If we keep using the "smooth line" method to judge them, we are getting a false sense of security. By switching to the "stepwise" way of measuring and using the "logprobs" trick to get more detailed answers, we can finally trust these AI classifiers to tell us when they are truly confident and when they are just guessing. It's a small change in how we listen, but it makes a huge difference in how well we can use these digital helpers.