💻 computer science

Uncertainty Quantification for LLM Function-Calling

This paper presents the first evaluation of Uncertainty Quantification (UQ) methods for LLM function-calling, discovering that while multi-sample methods offer little advantage over single-sample ones in this setting, their performance can be significantly improved by leveraging the structural and semantic specificities of function-call outputs.

Original authors: Zihuiwen Ye, Lukas Aichberger, Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal, Arno Blaas, Adam Golinski

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihuiwen Ye, Lukas Aichberger, Michael Kirchhof, Sinead Williamson, Luca Zappella, Yarin Gal, Arno Blaas, Adam Golinski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a highly advanced robotic butler. This butler is amazing at following instructions, but it has one dangerous quirk: it doesn't just talk; it acts. If you ask it to "clean the kitchen," it might accidentally throw your expensive heirloom china into the dishwasher because it wasn't quite sure which dishes were fragile.

This paper is about teaching that butler how to say, "Wait, I’m not 100% sure I understood that correctly. Can you double-check before I proceed?"

In the world of AI, this is called Uncertainty Quantification (UQ) for Function-Calling.

The Problem: The "Confident Idiot"

Most Large Language Models (like ChatGPT) are designed to be helpful and assertive. When you ask an AI to perform a task—like "Transfer $500 to Mom"—it generates a specific piece of code (a "function call") to make it happen.

The problem is that AI can be a "confident idiot." It might generate a command that looks perfectly correct but has a tiny, catastrophic error (like transferring $5,000 instead of $500). Because the AI is so confident in its own mistake, it doesn't realize it's about to cause a disaster.

The Experiment: Testing the "Gut Feeling"

The researchers wanted to see if we could measure the AI's "gut feeling" (its uncertainty) to prevent these mistakes. They tested two main ways of measuring this:

The "Single Guess" Method (The Speedster): You ask the AI once, and you look at how "hesitant" it sounds while typing. If it's choosing words with very low probability, it's like a person stuttering—it’s a sign of doubt.
The "Multiple Brainstorming" Method (The Committee): You ask the AI the same question 10 different times. If all 10 answers are slightly different, the "committee" is in disagreement, which means the AI is likely confused.

The Surprising Discovery:
You would think the "Committee" (the more expensive, complex method) would be smarter. But the researchers found that for these specific technical tasks, the "Speedster" (Single Guess) was actually just as good, if not better! The complex brainstorming didn't add much value because the AI's mistakes in coding are often very specific and don't show up as much in "disagreement" between samples.

The Solution: The "Highlighter" Trick

The researchers realized that not all words in a command are equal.

Imagine you are following a recipe. If the recipe says, "Add 2 tablespoons of salt," and you aren't sure if it's "tablespoons" or "teaspoons," that's a big deal. But if you aren't sure if the word "Add" should be capitalized or not, it doesn't matter at all.

The researchers created a new method called SMT (Semantically Meaningful Tokens). Instead of looking at the AI's confidence in every single character (like commas, brackets, and spaces), they taught the system to act like a highlighter. It ignores the "boring" syntax (the commas and brackets) and only measures the AI's confidence on the "high-stakes" words—the actual names of the functions, the numbers, and the specific names of people or places.

The Result: A Safer Butler

By focusing only on the "high-stakes" parts of the command, the AI became much better at flagging its own mistakes.

In short: This paper provides a blueprint for building "guardrails" around AI. It tells us that instead of making AI do massive amounts of extra thinking, we can make it much safer simply by teaching it to pay closer attention to the parts of its instructions that actually matter.

Technical Summary: Uncertainty Quantification for LLM Function-Calling

1. Problem Statement

As Large Language Models (LLMs) transition from text generators to autonomous agents, they increasingly rely on Function-Calling (FC) to interact with external environments (e.g., executing code, making financial transactions, or modifying databases). Unlike standard natural language generation, errors in FC can lead to irreversible and high-stakes consequences, such as data corruption or financial loss.

While Uncertainty Quantification (UQ) is a known method to mitigate these risks—allowing a system to "abstain" or ask for human help when confidence is low—there has been a lack of systematic evaluation of UQ methods specifically tailored to the structured, syntax-heavy nature of function calling. Existing UQ methods are largely optimized for natural language Q&A, and it is unclear if they translate effectively to the FC paradigm.

2. Methodology

The authors conduct the first comprehensive evaluation of UQ methods in the FC setting using a benchmark derived from the Berkeley Function Calling Leaderboard (BFCL).

A. Benchmarking Framework:

Datasets: The authors use Python-based tasks including Simple (one function), Multiple (choosing between functions), Parallel (calling multiple functions), and Parallel-Multiple (complex combinations). They also include Irrelevance tasks to test if models can recognize when a task is impossible to solve with available tools.
Evaluation Metric: They use AUROC (Area Under the Receiver Operating Characteristic curve) to measure how well uncertainty scores discriminate between correct and incorrect function calls. They also evaluate calibration using smoothECE (Expected Calibration Error).

B. UQ Methods Evaluated:

Single-Sample Methods: Based on a single greedy decoding, aggregating token-level negative log-likelihoods (NLL) via MAX (highest NLL), AVG (average NLL), and G-NLL (sum of NLLs).
Multi-Sample Methods: Sampling multiple sequences to estimate entropy, including Predictive Entropy (PE) and Semantic Entropy (SE) (which clusters semantically equivalent responses).
Prompting-based: P(True), which asks the model to assess the correctness of a candidate answer.

C. Proposed Adaptations:
The authors propose two novel variants to leverage the structural properties of FC:

AST-based Clustering (for Multi-sample): Instead of exact string matching, they use Abstract Syntax Trees (AST) to cluster outputs, treating function arguments as permutation-invariant dictionaries.
Semantically Meaningful Tokens (SMT) (for Single-sample): They identify tokens that carry actual decision weight (e.g., function names, parameter names, and values) and exclude syntactically required tokens (e.g., =, [, ]) from the uncertainty calculation to prevent "noise" from distorting the score.

3. Key Contributions

First FC-UQ Benchmark: Established a rigorous evaluation framework for uncertainty in tool-use scenarios.
Discovery of Method Divergence: Demonstrated that the "gold standard" multi-sample methods for natural language (like Semantic Entropy) do not provide a clear advantage over simple single-sample methods in the FC setting.
Structural Adaptation: Developed SE-AST and G-NLL-SMT, showing that tailoring UQ to the syntax and semantics of code/API calls improves performance.

4. Key Results

Single-Sample Dominance: In the FC setting, G-NLL (Greedy Negative Log-Likelihood) generally outperformed more complex multi-sample methods. This is attributed to the fact that LLMs in FC tasks assign very high probabilities ( $\approx 1.0$ ) to most tokens, causing multi-sample entropy to collapse into a single semantic cluster even when the call is incorrect.
Effectiveness of SMT: The G-NLL-SMT variant consistently outperformed the standard G-NLL across almost all task combinations and improved calibration (lower smoothECE).
AST Improvement: Using AST for clustering in multi-sample methods improved AUROC in most tasks compared to exact string matching, though it still did not surpass single-sample methods.
Risk-Coverage Gains: The authors demonstrated that using UQ for selective prediction (abstaining from low-confidence calls) significantly boosts accuracy. For example, in "Parallel-Multiple" tasks, abstaining from the top 30% most uncertain calls increased accuracy from 79% to 86%.

5. Significance

This work is highly significant for the development of reliable and safe autonomous AI agents. It provides a roadmap for engineers to implement "guardrails" in tool-use systems. The finding that simple, computationally efficient single-sample methods (specifically those focused on semantically meaningful tokens) are superior to expensive multi-sample methods is particularly important for on-device AI and real-time applications where latency and throughput are critical constraints.