🤖 machine learning

LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling

This paper proposes a metacognitive harness that leverages large language models' latent self-monitoring signals (feeling-of-knowing and judgment-of-learning) to dynamically control test-time reasoning, significantly improving performance across text, code, and multimodal benchmarks without requiring model fine-tuning.

Original authors: Qi Cao, Yufan Wang, Peijia Qin, Shuhao Zhang, Pengtao Xie

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Cao, Yufan Wang, Peijia Qin, Shuhao Zhang, Pengtao Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Overconfident Student"

Imagine a brilliant student taking a difficult exam. This student has a hidden superpower: they can accurately sense how well they are doing.

Before answering: They can feel a "gut instinct" about whether they know the material (called FOK or Feeling of Knowing).
After answering: They can look at their work and honestly judge, "I'm pretty sure this is right" or "I think I messed this up" (called JOL or Judgment of Learning).

The Problem: Even though this student has these accurate feelings, they don't use them. If they feel unsure, they just write down an answer and move on. If they feel confident, they stop immediately. They don't say, "Wait, I'm unsure, let me think harder," or "I'm confident, I can skip the next question." They just keep plowing through at the same speed, regardless of how hard the question is.

The paper argues that current Large Language Models (LLMs) are exactly like this student. They have the ability to know when they are right or wrong, but they don't act on that knowledge to change their behavior.

The Solution: The "Metacognitive Harness"

The researchers built a "harness" (like a control system or a coach) that forces the model to actually use these feelings. They took inspiration from a psychological theory called Nelson–Narens, which separates "monitoring" (checking your work) from "control" (deciding what to do next).

Here is how their system works, step-by-step:

1. The "Gut Check" (Pre-Solve)

Before the model tries to solve a problem, the harness asks: "How likely are you to get this right?"

Analogy: It's like a driver checking the weather before a trip. If the driver says, "It looks like a storm," the harness knows to prepare for a difficult drive.

2. The "Self-Grade" (Post-Solve)

After the model gives an answer, the harness asks: "How confident are you that this answer is correct?"

Analogy: It's like a chef tasting a dish before serving it. If the chef says, "This tastes a bit off," the harness knows to send it back to the kitchen.

3. The "Coach's Decision" (The Control Loop)

This is the magic part. The harness doesn't just listen; it makes a decision based on a learned rule (a specific "decision boundary" trained on a small set of practice problems).

If the model is confident: The harness says, "Great, you're done. Move on." (Saving time and money).
If the model is unsure: The harness says, "Stop! You said you weren't sure. Let's try again, but this time, look at why you were unsure and try a different approach."
The "Retry" Trick: When the model retries, the harness gives it a "cheat sheet" of its own doubts (e.g., "You were unsure because the numbers didn't add up") but hides the actual wrong answer it just wrote. This forces the model to try a fresh path instead of just repeating the same mistake.

4. The "Final Judge" (Aggregation)

If the model tries multiple times, a final "judge" looks at all the different attempts and picks the best one. Crucially, this judge ignores the confidence scores and just looks at the logic and the answer itself.

Why? Because the paper found that confidence scores are great for deciding whether to try again, but they are bad at deciding which of two similar attempts is better.

The Results: "Steering" the Engine

The researchers tested this on a fixed version of the Claude Sonnet-4.6 model. They didn't change the model's brain (no retraining); they just added this "harness" to control how it thinks.

The Result: The model got significantly smarter. On a mix of hard math, coding, and visual puzzles, the accuracy jumped from 48.3% to 56.9%.
The Comparison: This simple "harness" made the model perform better than the top-ranked models on public leaderboards, even though those other models were likely more powerful to begin with.
The Efficiency: The system was smart about spending money. It spent a lot of effort on the hard questions (where the model was unsure) and very little effort on the easy questions (where the model was confident).

Key Takeaways in Plain English

LLMs already "know" they are unsure: They aren't blind. They can tell you when they are guessing.
They just need a "Steering Wheel": Without a system to force them to act on that knowledge, they waste time on easy questions and give up too quickly on hard ones.
It's about Control, not Smarts: You don't need to make the model "smarter" (which is expensive and hard). You just need to give it a better way to manage its own thinking process.
The "Diagnosis" is Crucial: Before using this system, you have to check if the specific model is actually good at sensing its own confidence. Some models are "good at guessing" but "bad at knowing they are guessing." The harness only works on models that pass this "diagnosis."

In summary: The paper shows that if you give a smart AI a "coach" that tells it when to stop, when to retry, and when to try a new angle based on its own feelings of confidence, it can solve problems much better without needing to be retrained. It turns a passive, overconfident student into an active, self-correcting learner.

Technical Summary: LLMs Know When They Know, but Do Not Act on It

Problem Statement

Large Language Models (LLMs) frequently exhibit latent metacognitive abilities: they can estimate their likelihood of success before solving a problem (Feeling of Knowing, FOK) and judge the correctness of their answers after solving (Judgment of Learning, JOL). However, current research largely treats these signals as passive diagnostic metrics to be measured or calibrated, rather than as active control mechanisms.

The core problem identified is a disconnect between monitoring and control. While LLMs can report confidence scores that correlate with actual correctness, they do not inherently use these scores to adapt their reasoning behavior. High-confidence examples do not consistently receive less reasoning effort, nor do low-confidence examples reliably trigger retries or additional verification. Consequently, the potential for "test-time scaling"—allocating more compute to difficult problems and less to easy ones—remains untapped because models lack an explicit interface to act on their own self-knowledge.

Methodology: The Metacognitive Harness

Inspired by the Nelson–Narens theory of metamemory, which separates monitoring (meta-level) from control (object-level), the authors propose a Metacognitive Harness. This framework transforms self-monitoring signals into an inference-time controller that regulates reasoning without modifying the base model's parameters.

1. Signal Elicitation

For each problem, the harness elicits two scalar signals in the range $[0, 1]$ :

Feeling of Knowing (FOK): A pre-solve estimate of whether the model knows how to solve the problem. The prompt explicitly forbids computation or derivation at this stage to capture pure intuition.
Judgment of Learning (JOL): A post-solve estimate of the correctness of the generated answer.

2. Metacognitive Diagnosis

Before deployment, a lightweight diagnosis is performed on a small, disjoint "anchor set" (100 examples) to determine if a model's signals are usable for control. This stage evaluates two axes:

Discrimination: Measured by AUROC, assessing whether higher scores rank correct attempts above incorrect ones.
Calibration: Measured by Expected Calibration Error (ECE), assessing whether confidence magnitudes match empirical accuracy.

Based on these metrics, a model-specific decision function, $g_m(FOK, JOL)$ , is learned using a Support Vector Machine (SVM). This function defines a decision boundary to determine when to trust a current attempt and when to allocate additional compute.

3. The Control Loop

The harness operates as a selective test-time scaling loop:

Solve: The model generates a reasoning trace and an answer.
Judge: The model outputs a JOL score and a natural-language reason for that confidence.
Decide: The learned decision function $g_m$ $g_{m}$ evaluates the pair $(FOK, JOL)$.
- If the decision is to Stop, the current answer is accepted (or passed to an aggregator if multiple attempts exist).
- If the decision is to Retry, the model attempts the problem again.
Context Management: Crucially, retry attempts are guided by a compact metacognitive history (previous answers, JOL scores, and JOL reasons) rather than full previous reasoning traces. This prevents the model from being anchored to flawed derivations while still providing feedback on why the previous attempt was uncertain.
Aggregation: If multiple attempts are generated, a separate verifier-style aggregator selects the best answer based on reasoning traces and answers, explicitly excluding FOK/JOL signals to avoid biasing the selection with potentially misleading self-confidence cues.

Key Contributions

Metacognitive Harness: A novel control interface that converts latent self-monitoring signals (FOK/JOL) into explicit reasoning control (trust, retry, stop, aggregate), grounded in cognitive psychology.
Lightweight Diagnosis Protocol: A method to evaluate whether a specific LLM exposes discriminative and calibrated metacognitive signals suitable for control, preventing the application of the harness to models where such signals are absent or uncalibrated.
Parameter-Free Improvement: Demonstration that a fixed base model (Claude Sonnet-4.6) can achieve state-of-the-art performance across text, code, and multimodal reasoning benchmarks solely through inference-time control, without fine-tuning or parameter updates.

Results

The harness was evaluated on three public benchmarks: HLE-Verified (STEM), LiveCodeBench v6 (Code), and R-Bench-V (Multimodal).

Performance Gains: On a pooled set of 1,859 examples, the harness improved the fixed Claude Sonnet-4.6 base model from 48.3% accuracy (Pass@1) to 56.9%.
Benchmark Specifics:
- HLE-Verified: Improved from 48.0 to 60.0 (+12.0).
- LiveCodeBench v6: Improved from 74.3 to 84.3 (+10.0), with a massive +19.6 gain on the "Hard" split.
- R-Bench-V: Improved from 36.1 to 41.1 (+5.0).
Comparison to Baselines: The harness outperformed both vertical scaling methods (e.g., Self-Refine, Budget Forcing) and parallel scaling methods (e.g., Verifier Reranking, Aggregator) under matched or comparable inference budgets. It achieved the highest rank (#1) in efficiency and accuracy trade-offs.
Ablation Insights:
- Gains depend on the combination of FOK and JOL; removing either degrades performance.
- Learned SVM controllers outperform hand-crafted thresholds.
- Directed retry (using metacognitive feedback) significantly outperforms random retry or fixed-budget sampling.
- Context management is critical: exposing full previous reasoning chains leads to anchoring errors, while compact metacognitive feedback enables effective correction.

Significance and Claims

The paper claims that strong LLMs already possess useful metacognitive abilities but lack the explicit "harness" to act on them during reasoning. The significance lies in shifting the paradigm from treating confidence as a passive diagnostic to using it as an active control signal.

The authors emphasize that:

Metacognition is not universal: Not all models are "harnessable." The diagnosis step is essential to identify models with signals that are both discriminative and calibrated.
Adaptive Effort Allocation: The harness successfully converts confidence into an adaptive policy, allocating more reasoning attempts to uncertain problems and stopping early on confident ones, a behavior not exhibited by the base model in isolation.
Distinction from Aggregation: Confidence signals are effective for deciding whether to retry (across problems) but are less effective for selecting the best answer among multiple attempts within a single problem, necessitating a separation of control and aggregation stages.

The work suggests that "steering" agents via explicit harness engineering is a viable path to unlocking latent reasoning capabilities in existing models, potentially offering a more efficient alternative to scaling model parameters or training on specific benchmarks.