💻 computer science

HugSelect: An Explainable Multi-Criteria Decision-Support Framework for foundation-model selection

This paper introduces HugSelect, an explainable multi-criteria decision-support framework that constructs a comprehensive knowledge base of over 71,000 foundation models to provide auditable, transparent, and high-quality recommendations by prioritizing functional capabilities and community-perceived quality over simple popularity metrics.

Original authors: Alireza Joonbakhsh (Shiraz University), Arda Canser Adalı (Utrecht University), Slinger Jansen (Utrecht University), Farshad Khunjush (Shiraz University), Siamak Farshidi (Wageningen University,Resear

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Alireza Joonbakhsh (Shiraz University), Arda Canser Adalı (Utrecht University), Slinger Jansen (Utrecht University), Farshad Khunjush (Shiraz University), Siamak Farshidi (Wageningen University,Research)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, chaotic library where the shelves stretch up into the clouds and disappear into the fog. This isn't a library of books, but of "foundation models"—the giant, reusable brain-circuits that power everything from chatbots to medical diagnostic tools. In the world of software engineering, these models are like the engines of a car; you can't just pick one because it looks cool or because it's the most popular brand. You need to know if it fits your specific chassis, if it runs on the fuel you have, and if it won't break down when you hit a bump.

The problem is that this library is growing so fast that no human can read every label. Usually, people just grab the most downloaded item or ask a super-smart robot (an AI) for advice. But asking a robot is like asking a magician for a recipe; they might give you a delicious answer, but you have no idea how they made it, and sometimes they might just make things up. This paper, written by a team of researchers, asks a simple but tricky question: How do we pick the right engine for our car when the library is too big to read and the magicians are a bit too mysterious? They propose a new way to choose that doesn't just guess, but actually explains why it picked what it picked, turning a magical guess into a clear, auditable checklist.

The Problem: The "Popularity Contest" Trap

Right now, if you want to find a foundation model on platforms like Hugging Face, it's a bit like shopping for a new phone based only on how many people have bought it. You see a list sorted by "downloads" or "likes." But popularity isn't the same as performance. Just because a model is famous doesn't mean it can do the specific job you need, like translating ancient languages or running on a tiny laptop.

The authors argue that picking a model shouldn't be a simple keyword search or a popularity contest. It's a complex software engineering decision. You have to balance functional needs (can it write code?), operational constraints (does it fit in my memory?), and quality concerns (is it reliable?). Relying only on popularity or asking a "black box" AI for advice leaves you in the dark about why a model was chosen.

The Solution: HugSelect, the "Explainable Librarian"

Enter HugSelect. Think of HugSelect not as a magic 8-ball, but as a super-organized, hyper-logical librarian who has read every single book in the library and taken detailed notes on every page.

Here is how HugSelect works, using a fun analogy:

Imagine you are looking for a specific type of car. You tell the librarian, "I need a car that can drive on snow, has a sunroof, and costs less than $20,000."

Old Way (Popularity): The librarian points to the most popular car in the showroom, even if it's a convertible with no heater.
Old Way (Magic AI): The librarian says, "I think this car is great!" but can't tell you why or what the engine specs are.
HugSelect Way: The librarian pulls out a giant spreadsheet. They check the "Snow" column, the "Sunroof" column, and the "Price" column for every single car. They give each car a score based on how well it matches your needs. Then, they hand you a report that says: "Car A got 90 points because it has a sunroof and is cheap, but it lost points because it's not great in snow. Car B got 85 points because it's perfect for snow, but it's expensive."

HugSelect does exactly this for AI models. It builds a massive "knowledge base" of 71,274 models. It doesn't just look at the title; it reads the model's description, checks the code, and even scans thousands of community discussions (like Reddit threads and Q&A sites) to see what real people think about the model's reliability and speed.

How It Works: The Three Ingredients

To build its "scorecard" for every model, HugSelect mixes three types of ingredients:

Metadata: The hard facts, like the license type (is it free to use?) and the file size.
Functional Features: What the model can actually do. Did the description say it can handle images? Can it reason through math problems? HugSelect reads the text to find these capabilities.
Perceived Quality: This is the "gossip" column. It analyzes what the community says. If people keep complaining that a model crashes often, HugSelect marks it down on "Reliability." If people say it's super fast, it gets a boost on "Performance."

Once it has all this data, it uses a mathematical method called Weighted Sum Model (WSM). Imagine you are grading a student. If "Math" is 50% of the grade and "Art" is 10%, HugSelect adds up the scores based on what you care about most. If you say, "I don't care about speed, I just need it to be reliable," HugSelect adjusts the weights and re-ranks the list instantly.

What They Found: The Results

The researchers tested HugSelect to see if it actually works. They didn't just guess; they ran it through a gauntlet of tests.

The Reading Test: They checked if HugSelect could correctly read the model descriptions and community reviews. It got it right about 80% of the time for features and 84% of the time for quality attributes. That's pretty good for a robot reading messy human text!
The Showdown: They compared HugSelect against four famous commercial AI systems (like ChatGPT, Claude, and Gemini). They gave all of them 44 different scenarios, like "Find a model for a medical Q&A bot."
- HugSelect found the right family of models 91% of the time.
- It found the exact model 61% of the time.
- This was competitive with the big commercial AIs (some were slightly better at finding the exact model, some slightly worse), but HugSelect had one huge advantage: Transparency. The commercial AIs just gave an answer. HugSelect gave the answer plus the math behind it, showing exactly which features made the model win or lose points.

The "Why It Matters" Factor

The most important thing HugSelect does is make the decision auditable. In software engineering, you can't just say, "The AI told me to use this." You need to know why so you can explain it to your boss, your client, or a safety inspector.

The researchers also did a "user study" with 10 people who know about AI. These people found HugSelect useful and easy to use. They liked that they could see the trade-offs. For example, they could see that Model A was faster but less reliable, while Model B was slower but rock-solid. This helps humans make better decisions rather than blindly trusting a black box.

The Bottom Line

This paper suggests that we need to stop treating AI model selection like a game of chance or a popularity contest. By building a system that reads the fine print, listens to the community, and explains its math, HugSelect offers a way to pick the right tool for the job with confidence. It doesn't claim to be perfect—the researchers admit that sometimes the community feedback is messy, and the "ground truth" of what the "best" model is can be tricky. But it proves that with the right mix of data and clear logic, we can turn the chaotic library of AI models into a well-organized, explainable toolbox.

In short: HugSelect is the librarian who doesn't just point to the most popular book, but opens it up, highlights the best parts, and tells you exactly why it's the right choice for your story.

Technical Summary: HugSelect – An Explainable Multi-Criteria Decision-Support Framework for Foundation-Model Selection

1. Problem Statement

The selection of foundation models has evolved from a research activity into a critical software engineering decision, as these models serve as reusable components in AI-enabled systems. However, current selection mechanisms face significant limitations:

Popularity and Metadata Filters: Relying on download counts or tags provides only surface-level representations, failing to capture functional capabilities or operational constraints.
Keyword Retrieval: Lexical matching struggles with heterogeneous information sources and cannot differentiate between models with similar descriptions but distinct functional capabilities.
LLM-Based Recommenders: While capable of interpreting complex queries, conversational LLMs often operate as "black boxes," lacking transparent, auditable reasoning and being susceptible to hallucinations.

The core problem is the lack of an explainable, repository-scale decision-support system that treats foundation-model selection as an explicit, auditable software-component selection task involving functional requirements, non-functional quality concerns, and operational constraints.

2. Methodology: The HugSelect Framework

HugSelect is a data-driven framework grounded in Design Science Research (DSR) and Multi-Criteria Decision Making (MCDM). It operationalizes model selection through a unified pipeline that integrates heterogeneous data sources into a knowledge base and applies a transparent ranking algorithm.

A. Data Integration and Knowledge Base Construction
The system constructs a knowledge base of 71,274 Hugging Face models by synthesizing three complementary evidence sources:

Repository Metadata ( $F_M$ ): Structured attributes (task tags, license, parameter size, download counts).
Functional Features ( $F_F$ ): Capabilities extracted from unstructured model cards and README files using NLP techniques (noun-phrase filtering, dependency parsing). These are categorized into modality, task, and model family.
Perceived Quality Attributes ( $F_Q$ ): Non-functional signals derived from community discussions (Hugging Face, Reddit, Stack Overflow). These are mapped to ISO/IEC 25010 quality characteristics (e.g., reliability, performance efficiency) using sentiment analysis and LLM-based classification.

B. Decision Engine and Explainability

Intent Extraction: User natural-language queries are processed to extract structured criteria and weights. The system employs a MoSCoW prioritization scheme (Must-have, Should-have, Could-have, Won't-have) to assign importance weights to criteria.
Ranking Algorithm: The framework utilizes the Weighted Sum Model (WSM/SAW). For a candidate model $m_i$ and query $q$ , the score is calculated as:
$R(m_i, q) = \sum_{j=1}^{k} w_j \cdot s_{ij}$
Where $w_j$ is the criterion weight and $s_{ij}$ is the normalized score.
Explainability: Unlike black-box recommenders, HugSelect decomposes the final score into criterion-level contributions ( $w_j \cdot s_{ij}$ ), links scores back to source evidence (e.g., specific review snippets), and visualizes trade-offs between model variants via a knowledge graph.

3. Key Contributions

The paper claims four primary contributions:

Repository-Scale Decision Support: A formulation of foundation-model selection as an auditable software-component decision, moving beyond simple search or popularity ranking.
Integrated Evidence-Synthesis Pipeline: An automated pipeline that harvests metadata, functional capabilities, and community-derived quality signals into a unified knowledge base of over 71,000 models.
Explainable Decision Engine: A WSM/SAW-based engine that maps user requirements to structured criteria and decomposes recommendations into inspectable, criterion-level score contributions.
Comprehensive Empirical Evaluation: A multi-faceted evaluation including pipeline validation, comparative case studies against commercial baselines, ablation analysis, and a practitioner user study.

4. Evaluation Results

The framework was evaluated through three complementary approaches:

Pipeline Validation (RQ1):
- Functional Feature Extraction: Achieved an F1-score of 0.801 and an IoU of 0.668 against a human-validated ground truth.
- Quality Attribute Mapping: Achieved 0.84 accuracy in mapping review snippets to ISO/IEC 25010 quality attributes.
- Finding: Automated extraction is effective at scale, though challenges remain with heterogeneous developer documentation.
Comparative Case Studies (RQ2):
- Evaluated against four commercial LLM baselines (ChatGPT, Claude, Gemini, Perplexity) across 44 literature-derived scenarios.
- Model-Level Coverage@10: HugSelect achieved 0.61, outperforming Gemini (0.39) and Perplexity (0.52), and performing comparably to Claude (0.64) and ChatGPT (0.52).
- Family-Level Coverage@10: HugSelect achieved 0.91, indicating strong ability to identify relevant model families even when exact model IDs differ.
- Statistical Significance: HugSelect significantly outperformed Gemini ( $p < 0.05$ ) but showed no statistically significant difference compared to ChatGPT or Claude in overall ranking quality.
- Qualitative Finding: Unlike LLM baselines which often exhibited popularity bias, HugSelect provided structured, traceable reasoning.
Ablation Study:
- Removing functional features caused the largest drop in model-level coverage (0.61 $\to$ 0.23), identifying them as the primary driver of exact-model retrieval.
- Removing quality attributes reduced family-level discrimination (0.91 $\to$ 0.77).
- Metadata-only approaches yielded the lowest performance (0.09 model-level coverage).
User Study (RQ3):
- An exploratory study with 10 participants based on the Technology Acceptance Model (TAM).
- Participants rated the system highly for Perceived Usefulness (Mean = 1.62) and Perceived Ease of Use (Mean = 1.48), suggesting the structured, transparent approach is intuitive and valuable.

5. Significance and Claims

The paper positions HugSelect not as a system that always identifies the single "optimal" model, but as a tool that makes the selection rationale explicit and auditable.

Transparency over Opacity: The primary significance lies in shifting from opaque conversational advice to a traceable decision process where users can inspect why a model is recommended and which trade-offs (e.g., performance vs. license) drove the ranking.
Software Engineering Perspective: It reframes foundation-model selection as a standard software component selection problem, integrating non-functional requirements (licensing, deployment constraints) alongside functional capabilities.
Feasibility: The results provide preliminary evidence that repository mining and MCDM can be successfully combined to support transparent, large-scale model selection, offering a practical alternative to purely popularity-based or black-box AI recommendations.

The authors acknowledge limitations, including the use of literature-reported choices as proxy ground truth, the reliance on perceived rather than objective quality metrics, and the exploratory nature of the user study. They conclude that while HugSelect is competitive with commercial baselines in retrieval quality, its distinct value is the provision of stable, traceable, and inspectable reasoning for software engineers.