🤖 AI

VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection

The paper introduces VecCISC, a lightweight framework that reduces the computational cost of Confidence-Informed Self-Consistency by using semantic similarity to filter redundant or hallucinated reasoning traces before evaluation, achieving up to 47% token savings while maintaining or improving accuracy across diverse reasoning benchmarks.

Original authors: James Petullo, Sonny George, Dylan Cashman, Nianwen Xue

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: James Petullo, Sonny George, Dylan Cashman, Nianwen Xue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky riddle. You ask a super-smart but sometimes chatty robot (an AI) to solve it. To make sure the robot gets it right, you ask it to try solving the riddle 20 times.

The Old Way (Self-Consistency):
You take all 20 answers and just pick the one that appears most often. If 15 robots say "The answer is Bowling," and 5 say "The answer is Chess," you go with Bowling. This works, but it's slow because you have to wait for 20 full answers.

The "Think Twice" Upgrade (CISC):
Researchers realized that just counting votes isn't enough. Sometimes the robot gives a wrong answer with a very confident-sounding explanation, or a right answer with a shaky one. So, they added a "Judge" robot.

The first robot gives 20 answers with their reasoning.
The Judge robot reads every single one of those 20 reasoning stories and gives them a "confidence score" (like a grade from 0 to 100).
You pick the answer with the highest total score.

The Problem:
This is great for accuracy, but it's incredibly expensive. Asking the Judge to read 20 long stories for every single question is like hiring a team of 20 editors to review 20 drafts of a letter. It takes a lot of time and money (computing power). Also, the first robot sometimes gets confused and writes nonsense (hallucinations) or repeats the same sentence over and over (degenerate traces). The Judge wastes time reading these garbage drafts too.

The New Solution (VecCISC):
The authors of this paper, James Petullo and his team, came up with a clever shortcut called VecCISC. Think of it as a "Smart Sorter" that sits between the first robot and the Judge.

Here is how it works, using a simple analogy:

The Grouping: The first robot still writes 20 stories.
The "Vibe Check" (Clustering): Instead of sending all 20 stories to the Judge, the Smart Sorter looks at them and groups them by "vibe" or meaning.
- Imagine the robot wrote 15 stories that all say, "The movie is about bowling," but they used slightly different words. The Sorter sees these are essentially the same story.
- It puts those 15 into one pile.
- It puts the 3 stories that say "It's about chess" into another pile.
- It finds the 2 stories that are just gibberish or broken code and puts them in a "trash" pile.
The Representative: From the "Bowling" pile, the Sorter picks just one story that is the most "average" or perfect example of that group. It ignores the other 14 because they are just copies. It ignores the gibberish pile entirely because it's clearly broken.
The Judge's Job: Now, instead of the Judge reading 20 stories, it only has to read one story from the "Bowling" pile and one from the "Chess" pile.

The Results:
By doing this, the team found that:

They saved a massive amount of money and time: They reduced the total work (tokens) by 47%. It's like cutting the bill in half.
They didn't lose accuracy: In fact, because the Judge wasn't distracted by the garbage or repetitive stories, the final answer was often more accurate than before.
It works everywhere: They tested this on math, science, biology, and general knowledge questions, and it worked well for all of them.

In a Nutshell:
VecCISC is like a smart librarian who realizes that if you have 20 copies of the same book, you only need to read one of them to know what it's about. By filtering out the duplicates and the trash before the expensive "Judge" reads them, the whole process becomes much cheaper and faster, without sacrificing the quality of the final answer.

Technical Summary: VecCISC

Problem Statement

Scaling inference-time reasoning in Large Language Models (LLMs) has become a standard approach for improving performance on complex tasks. The foundational technique, Self-Consistency (SC), involves sampling multiple reasoning traces and selecting the most frequent answer. More recently, Confidence-Informed Self-Consistency (CISC) has demonstrated superior accuracy by employing a "think twice" paradigm: a critic LLM evaluates each candidate's reasoning trace to assign a confidence score, which is then used in a weighted majority vote.

However, CISC suffers from significant inference overhead and cost. Because the critic LLM must be called for every sampled reasoning trace, the token usage and latency effectively double compared to standard SC. Furthermore, naive CISC treats all samples equally, including hallucinated, degenerate, or semantically redundant traces, wasting computational resources on low-quality outputs that do not contribute to the final decision.

Methodology: VecCISC

To address these inefficiencies, the authors propose VecCISC, a lightweight, adaptive framework that reduces the number of reasoning traces requiring critic evaluation without sacrificing accuracy. The pipeline operates as follows:

Sampling: An LLM ( $LLM_{gen}$ ) generates $n$ reasoning trace-answer pairs for a given prompt.
Embedding Generation: Each reasoning trace is converted into a vector embedding using a pre-trained embedding model ($Emb$).
Grouping by Answer: Traces are grouped by their resulting answer ( $a$ ). This ensures that distinct candidate answers are preserved before clustering.
Clustering: Within each answer group, embeddings are clustered using algorithms such as KMeans or Hierarchical Agglomerative Clustering (HAC). The goal is to identify semantically equivalent traces, as well as isolate outliers (hallucinations or degenerate text). The number of clusters ( $K$ $K$ ) is determined via grid search on a holdout set.
- Note: The authors explicitly avoid DBSCAN due to the sensitivity of distance thresholds in high-dimensional embedding spaces.
Representative Selection: From each cluster, a single representative trace is selected. This is achieved by computing the centroid of the cluster's embeddings and selecting the trace closest to this centroid using cosine similarity. The authors argue that the trace closest to the centroid is the least likely to contain anomalous reasoning errors.
Confidence Scoring: Instead of evaluating all $n$ traces, the critic LLM ( $LLM_{critic}$ ) is only prompted with the representative traces from each cluster.
Final Aggregation: The confidence scores for the representatives are normalized (using softmax with a tunable temperature $T$ ) and used in a weighted majority vote to select the final answer.

Key Contributions

The paper outlines four primary contributions:

Novel Framework: Introduction of VecCISC, which utilizes reasoning trace clustering to eliminate redundant and low-quality samples from the pool analyzed by the critic, thereby reducing the cost of the "think twice" paradigm.
Empirical Validation: Extensive experiments across five diverse datasets (mathematics, chemistry, biology, commonsense reasoning, and humanities) and five different LLMs (ranging from 7B to 70B parameters) demonstrate that VecCISC reduces the total token budget by 47% while maintaining or exceeding the accuracy of standard CISC.
Comparative Analysis: The study compares clustering-based selection (KMeans and HAC) against random candidate selection ( $K$ samples), proving that semantic clustering is superior to random sampling for identifying ideal representatives.
Open Source: The authors commit to releasing their code to facilitate further research.

Results

The experiments were conducted on five datasets: AQuA-RAT, CommonsenseQA, ARC-Challenging, MMLU-Pro, and GPQA.

Cost Reduction: VecCISC significantly reduces the number of calls to the critic LLM. VecCISC + KMeans achieved an average 34.68% reduction in critic calls, while VecCISC + HAC achieved 30.2%. When considering the entire pipeline (including the initial sampling), the total token usage was reduced by 47%.
Token Efficiency: The "min-centroid" selection strategy (choosing the trace closest to the cluster center) consistently outperformed random trace selection, resulting in lower token consumption for the critic component (which accounts for ~77% of total tokens).
Accuracy: VecCISC variants consistently outperformed or matched standard Self-Consistency and CISC across nearly all model-dataset combinations. Notably, VecCISC + HAC achieved the best average results in most configurations, and in some cases, the reduced set of high-quality traces led to slight accuracy gains over CISC.
Ablation: The "VecCISC (random)" baseline, which selects $K$ traces randomly without clustering, significantly underperformed both CISC and SC, highlighting the necessity of semantic clustering to filter out noise and degenerate traces.

Significance and Claims

The paper claims that VecCISC offers an effective, lightweight augmentation to existing "think twice" pipelines. The authors assert that by filtering semantically equivalent and degenerate traces, the framework allows the critic LLM to focus on a smaller set of higher-quality reasoning traces. This approach mitigates the prohibitive inference costs associated with confidence calibration while preserving, and in some instances enhancing, model performance.

The authors remain modest regarding the generalization of their findings, noting that while they used a general-purpose embedding model ($text-embedding-3-small$) to ensure adaptability, specialized tasks might benefit from domain-specific, fine-tuned embedding models. Additionally, they acknowledge that the hyperparameters $K$ (cluster count) and $T$ (softmax temperature) currently require data-driven grid searches, suggesting that future work could explore adaptive methods for selecting these parameters without prior data.