💻 computer science

A Multi-Criteria Decision Framework for Aggregating Ranked ICD-10 Code Suggestions from Large Language Models

This study demonstrates that applying Multi-Criteria Decision Analysis rank aggregation methods, particularly Borda Count and Reciprocal Rank Fusion, to combine outputs from multiple Large Language Models significantly improves the accuracy and stability of automated ICD-10 clinical coding compared to using individual models or simple voting schemes.

Original authors: Ricardo da Silva Santos, Murilo Gleysson Gazzola, Paulo Marcelino Figueira, Adriana Gomes Luz, Rodolfo de Carvalho Pacagnella, Cristiano Torezzan

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ricardo da Silva Santos, Murilo Gleysson Gazzola, Paulo Marcelino Figueira, Adriana Gomes Luz, Rodolfo de Carvalho Pacagnella, Cristiano Torezzan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, messy puzzle where the pieces are medical diagnoses. In the world of healthcare, every patient's story needs to be translated into a specific code, like a secret language called ICD-10, so hospitals can track illnesses and get paid. But this language is huge, complicated, and full of tricky rules. Recently, super-smart computer programs called Large Language Models (LLMs) have learned to guess these codes by reading doctor's notes. Think of these models as a team of brilliant but slightly different detectives. Each detective looks at the same case file and comes up with their own list of suspects (the codes), ranked from "most likely" to "least likely." The problem is, sometimes Detective A thinks Suspect X is the culprit, while Detective B is convinced it's Suspect Y. If you only listen to one detective, you might get it wrong. This is where a field called Multi-Criteria Decision Analysis comes in. It's basically the art of taking many different opinions and mixing them together to find the single best answer, kind of like how a jury combines testimonies to reach a verdict. The big question is: how do you mix these lists of suspects so that the final group is smarter than any single detective on its own?

This paper tackles that exact question by treating each AI model as a separate "voter" in a grand election for the correct medical codes. The researchers took 1,117 real obstetric (pregnancy-related) medical notes written in Brazilian Portuguese and asked five different powerful AI models to generate a ranked list of possible codes for each note. Instead of picking the winner from just one model, they tested three different ways to combine all the lists into one "consensus" list. They tried a simple method called "Plurality Voting" (where the code that appears at the very top of the most lists wins), a more detailed method called "Borda Count" (where a code gets points based on how high it appears on every list, not just the top spot), and a method called "Reciprocal Rank Fusion" (which gives a big boost to codes that appear near the top but still counts those lower down).

The results were quite clear and surprising in their simplicity. The team found that combining the models almost always worked better than trusting the single best model alone. Specifically, the "Borda Count" method was the star of the show. When the researchers looked at the top 3 suggestions (a number they found to be the sweet spot for balancing accuracy and coverage), the Borda Count method improved the overall accuracy by about 20% at the broad category level and nearly 19% at the specific code level compared to the best individual AI. This suggests that listening to the "crowd" of AI models, especially by paying attention to where a code appears on everyone's list, creates a much more reliable decision than relying on a single expert.

However, the paper also rules out a few things. It found that the simplest method, Plurality Voting (just counting who is #1), didn't work as well as the more nuanced methods that looked at the whole ranking. It also showed that you can't just keep adding more suggestions to the list forever. While adding more codes helps you catch more true diagnoses (recall), it eventually drags down the accuracy because you start including too many wrong guesses (precision). The study suggests that the "golden number" is 3; going beyond that, the quality of the decision starts to drop. Furthermore, the authors note that simply giving more weight to the "strongest" models didn't help much; the unweighted, simple combination of all models was just as good as the complex weighted ones.

In short, the paper suggests that for clinical coding, the best strategy isn't to find the one perfect AI model, but to let a few different ones vote, using a system that respects their full rankings. This approach offers a practical, robust way to improve medical decision-making without needing to retrain the models or peek inside their "black boxes." The findings are based on real data and statistical testing with confidence intervals, showing that this method is a stable and effective way to handle the messy reality of medical coding.

Technical Summary: A Multi-Criteria Decision Framework for Aggregating Ranked ICD-10 Code Suggestions from Large Language Models

Problem Formulation
Automated clinical coding using the International Classification of Diseases, 10th Revision (ICD-10) remains a complex challenge due to the hierarchical nature of the taxonomy and the variability of medical terminology. While Large Language Models (LLMs) have demonstrated the ability to generate ranked lists of candidate ICD-10 codes from unstructured clinical text, individual models often exhibit instability and limited reliability when used in isolation. Different models may produce conflicting rankings for the same clinical note due to variations in training data and architectures.

This study addresses the decision problem of how to aggregate these disparate, ranked outputs into a single, reliable consensus. The authors formulate ICD-10 code assignment as a Multi-Criteria Decision Analysis (MCDA) problem, where each LLM acts as an independent decision agent providing a preference ordering over candidate codes. The core challenge involves determining the optimal aggregation strategy and the number of top-ranked codes ( $k$ ) to select to balance precision and recall in a practical decision-support context.

Methodology
The research utilizes a dataset of 1,117 obstetric clinical notes in Brazilian Portuguese, each annotated with ground-truth ICD-10 codes. The study employs five distinct LLMs (GPT-4o, DeepSeek-V3, Maritalk Sabiá-3.1, Gemini 1.5 Flash, and GPT-4o-Mini) as decision agents. These models generate ranked lists of candidate codes for each document.

The study evaluates three classical rank aggregation methods from voting theory and MCDA:

Plurality Voting: A "first-past-the-post" approach where a code receives a vote only if it appears in the top- $k$ of a model's ranking, regardless of its specific position within that range.
Borda Count: A positional scoring rule where a code at rank $r$ in a list of $n$ candidates receives $(n - r + 1)$ points. This method leverages the full ranking structure rather than just the top- $k$ presence.
Reciprocal Rank Fusion (RRF): A method that assigns weights based on the reciprocal of the rank position ( $1 / (k_0 + r)$ ), smoothing contributions from lower-ranked items while prioritizing top-ranked ones.

Performance is assessed using Micro-Precision, Micro-Recall, and Micro-F1 scores at two levels of granularity: the 3-character category level and the full "leaf" code level. Robustness is evaluated using bootstrap resampling with 95% confidence intervals. The study specifically analyzes the impact of varying $k$ (the number of selected codes) from 1 to 10.

Key Contributions
The paper makes three primary contributions:

Formalization: It formalizes the assignment of ICD-10 codes from multiple predictive models as a multi-criteria decision problem based on ranked alternatives.
Empirical Evaluation: It provides a comprehensive evaluation of classical rank aggregation methods (Plurality, Borda, RRF) across different code granularities in a non-English (Brazilian Portuguese) clinical setting.
Practical Implications: It demonstrates that simple, interpretable aggregation strategies can improve decision consistency without requiring model retraining or access to proprietary model internals.

Results
The empirical results indicate that rank-based aggregation methods consistently outperform the best individual model. Key findings include:

Optimal $k$ : Performance exhibits a non-monotonic behavior relative to $k$ . While increasing $k$ initially improves recall, it eventually degrades precision. The optimal balance for all methods was found at $k=3$ .
Superiority of Borda Count: The Borda Count method achieved the highest performance across both evaluation levels. At $k=3$ , it reached a Micro-F1 score of 0.4807 at the 3-character level (a 20.4% relative gain over the best individual model, GPT-4o-Mini) and 0.2112 at the full-code level (a 19.0% gain over DeepSeek-V3).
Comparison of Methods: Methods that incorporate full ranking information (Borda and RRF) consistently outperformed simple voting schemes (Plurality). However, weighted aggregation strategies did not yield statistically significant improvements over unweighted methods in this specific context.
Robustness: The improvements observed with aggregation were consistent across bootstrap confidence intervals, indicating statistical robustness.

Significance and Claims
The authors claim that this study demonstrates that rank aggregation provides a principled and effective approach for combining multiple imperfect decision agents in clinical coding tasks. The findings suggest that:

Ensemble Reliability: Aggregating outputs from multiple LLMs creates a more robust decision support system than relying on a single model, particularly in complex, multi-label scenarios.
Interpretability: Simple aggregation strategies like Borda Count can significantly enhance decision quality without the computational cost or complexity of fine-tuning models or accessing internal model parameters.
Human-in-the-Loop Suitability: The resulting consensus rankings are well-suited for human-in-the-loop workflows, where a short, high-quality list of candidate codes (e.g., top 3) can be efficiently reviewed and validated by expert coders, thereby improving both accuracy and efficiency in healthcare administration.

The study acknowledges limitations, including the use of a single clinical setting (obstetrics) and a focus on zero-shot prompting, suggesting future work should explore adaptive aggregation, reliability-aware weighting, and broader clinical domains.

Technical Summary: A Multi-Criteria Decision Framework for Aggregating Ranked ICD-10 Code Suggestions from Large Language Models

More like this