🤖 machine learning

Knee or ROC

This paper proposes using the knee method to determine accuracy thresholds for multi-class image detection with unknown population representations, addressing the limitations of traditional ROC curves in such scenarios.

Original authors: Veronica Wendt, Jacob Steiner, Byunggu Yu, Caleb Kelly, Justin Kim

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Veronica Wendt, Jacob Steiner, Byunggu Yu, Caleb Kelly, Justin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a busy art gallery. Your job is to look at pictures on the wall and tell the visitors what they are seeing.

In the past, the pictures were simple: a single painting of a cat or a single painting of a dog. The guard (the computer) was trained to say, "That's a cat!" or "That's a dog!" with high confidence. This is what most AI research has focused on so far.

The New Problem: The "Collage" Challenge
But now, the gallery owner is getting tricky. Instead of single paintings, they are hanging collages. A single frame might contain a cat, a dog, a car, and a tree all mixed together.

The old security guard gets confused. If the AI sees a collage, it might say, "I'm 40% sure it's a cat," but is that enough to call it a cat? What if it's actually a dog? The old rules for deciding "what is a cat" don't work well when there are multiple things in one picture, especially if the AI doesn't know exactly how many things are in there.

This paper is about teaching the security guard two new ways to make decisions when looking at these messy collages.

The Two New Strategies

The researchers tested two different ways to figure out the "cut-off point" (the threshold) for making a decision. Think of this as deciding: "How sure do I need to be before I shout, 'That's a cat!'?"

Strategy 1 & 2: The "Perfect Map" Approach (ROC)

Imagine you have a giant map of the territory. You want to find the perfect spot to build a fence so that you catch all the cats but let all the dogs through.

The Old Way: You build the fence while you are training, and once it's built, you can't move it. If the gallery changes its layout later, your fence is useless.
The New Way (Method 2): The researchers tried a "What If" game. They took the data and shuffled it around thousands of times, building a new fence for every single shuffle. They looked for the best possible fence that could be built from the data they had.
The Result: This "shuffling" method (Method 2) found a better fence than the standard map method. It gave them a very precise number (0.72) to say, "If the AI is 72% sure, we call it a match."

Strategy 3: The "Elbow" Method (Knee)

Sometimes, you don't have time to draw a map or shuffle data. You just need a quick rule of thumb.

The Analogy: Imagine you are walking up a steep hill that suddenly flattens out into a plateau. The point where the hill stops being steep and starts being flat is called the "knee" or the "elbow" of the curve.
How it works: The researchers looked at the list of confidence scores the AI gave for the collage. They sorted them from lowest to highest. They looked for that "elbow" point where the confidence suddenly jumps up.
The Discovery: They found a cool pattern:
- If the collage has a clear, big object (high confidence, like 35% or more), the "elbow" is easy to find. It's a reliable signal.
- If the collage is just a jumble of tiny, blurry things (low confidence), the "elbow" disappears. The line is too wiggly to trust.

The Big Takeaway

The paper is essentially saying: "We can't use the old rules for messy, multi-object pictures."

If you have time to calculate: Use the "Shuffling" method (Method 2). It's like trying every possible fence location to find the absolute best one. It's the most accurate but takes more computing power.
If you need a quick check: Use the "Elbow" method (Method 3). It's a fast way to see if the AI is actually seeing something clear or just guessing. If the "elbow" is strong, trust the AI. If the line is flat and wiggly, the AI is confused.

Why Does This Matter?

This is a stepping stone. Right now, AI is great at spotting one thing in a picture. But the real world is messy. We need AI that can look at a busy street scene and say, "I see a bus, a pedestrian, and a dog," even if it doesn't know exactly how many things are there beforehand.

This paper is the first step in teaching AI to be a better security guard for those messy, multi-object collages, using either a detailed map or a quick "elbow" check to make the right call.

1. Problem Statement

Current self-attention transformer models (specifically Compact Convolutional Transformers or CCT) have demonstrated high accuracy in image classification, but existing research is limited to single-class image detection with known population representations.

The Gap: Real-world scenarios often involve multi-class images (e.g., an image containing multiple objects) and unknown image populations (where the distribution of classes is not predefined).
Limitation of ROC: The traditional Receiver Operating Characteristic (ROC) curve determines accuracy thresholds during training. Once deployed in a live environment, these thresholds are static and cannot adapt to new or unknown data distributions, making them unsuitable for dynamic, multi-class detection.
Limitation of Knee Method: While the "knee" method (identifying the point of maximum curvature) allows for ad-hoc thresholding, its application to multi-class probability distributions requires novel adaptation.

2. Methodology

The authors propose three distinct methods to determine optimal accuracy thresholds for multi-class image detection using a Compact Convolutional Transformer (CCT-7/3x1) model trained on a custom dataset derived from CIFAR-10.

Dataset Construction

Created a multi-object dataset by randomly arranging four CIFAR-10 images into a 2x2 grid, resizing them to 32x32 pixels.
Generated 500 unique multi-object images for testing.

Proposed Methods

Method 1: CDF-Based ROC Generation
- Models True Positive Rate (TPR) and True Negative Rate (TNR) as overlapping normal distribution curves based on Softmax outputs.
- Uses the Cumulative Distribution Function (CDF) to calculate TPR and False Positive Rate (FPR) for arbitrary thresholds.
- Generates a novel ROC curve by incrementing the threshold and plotting the resulting TPR/FPR pairs.
Method 2: "Brute Force" Array Rearrangement
- Treats the problem as a "what-if" scenario. It iterates through threshold values (starting at 0.05 with 0.01 increments).
- For each threshold, it rearranges the original probability arrays (Positive and Negative) into four new arrays: True Positives (TP), False Positives (FP), False Negatives (FN), and True Negatives (TN).
- Generates a unique ROC curve for every threshold increment to find the configuration yielding the highest Area Under the Curve (AUC).
Method 3: The Knee Method (Ad-hoc)
- Designed for live environments where pre-calculated ROC curves are insufficient.
- Sorts the probability outputs of a multi-class image in ascending order.
- Plots these probabilities against their index to identify the "knee" (or elbow) point—the point of maximum curvature where the probability distribution shifts significantly.
- This method calculates thresholds on an ad-hoc basis for individual images during training, testing, or live deployment.

3. Key Contributions

Adaptation for Multi-Class Detection: The paper moves beyond single-class detection, proposing frameworks to handle images with multiple objects and unknown class distributions.
Novel ROC Formulations: Introduces two new ways to generate ROC curves for multi-class data: one using statistical distribution modeling (CDF) and another using iterative array rearrangement.
Ad-hoc Thresholding: Demonstrates the viability of the Knee method for dynamic thresholding in live environments, addressing the rigidity of traditional ROC curves.
Threshold Determination Algorithm: Proposes a specific calculation for the optimal threshold ( $T$ ) by finding the knee point of the best ROC curve, calculating the Euclidean distance to the $AUC=0.5$ line, and adding 0.5 to that distance.

4. Experimental Results

Method 1 vs. Method 2: Method 2 (Brute Force) outperformed Method 1. By iterating through threshold rearrangements, Method 2 produced a ROC curve with a larger AUC.
- Optimal Threshold: Using the KneeLocator function on the Method 2 curve, the optimal knee point was found at coordinates $(0.33, 0.64)$ . The calculated optimal threshold was $T = 0.72$ .
Method 3 (Knee Analysis):
- High Probability Images ( $\ge 0.35$ ): When the maximum probability in an image was $\ge 0.35$ , the knee values were consistent (typically between 6 and 8). In the test set, 58.8% of images met this criteria, and 92.5% of those had knee values between 6 and 8.
- Low Probability Images ( $< 0.35$ ): Knee values were inconsistent or non-existent, suggesting the method is less reliable for images with low confidence scores.
Performance Correlation: Method 2 is deemed more suitable for small multi-class objects (where precise thresholding is needed), while Method 3 is effective for larger, high-confidence multi-class objects.

5. Significance and Future Work

Significance: This research bridges the gap between the efficiency of transformer architectures (like CCT) and the complexity of real-world multi-object detection. It provides a mathematical framework for adapting accuracy thresholds dynamically, which is crucial for live deployment where data distributions are unknown.
Future Directions:
- Refining the ad-hoc knee method to handle cases where the number of classes in an input image is completely unknown.
- Integrating these thresholding techniques with object detection architectures like YOLO and Faster R-CNN to create state-of-the-art, lightweight multi-class detection systems.
- Further experimentation to improve the consistency of knee values for low-probability images.

Conclusion: The paper successfully demonstrates that while traditional ROC curves are static, combining iterative thresholding (Method 2) with ad-hoc knee analysis (Method 3) allows for robust, adaptable accuracy calculations in multi-class image classification scenarios.