💻 computer science

When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure

This paper identifies that indiscriminately fusing multiple foundation models often degrades performance due to redundancy and misalignment, and proposes KAGES, a label-aware, efficient greedy selection method that optimizes view-set composition to achieve superior downstream accuracy with a compact, task-aligned encoder subset.

Original authors: Yibo Liu, Bowen Jiang

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yibo Liu, Bowen Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern era of artificial intelligence, researchers have access to a vast library of pre-trained computer programs, often called foundation models. These are massive systems that have already learned to recognize patterns, understand language, or interpret images by studying enormous amounts of data. Think of them as expert tools that have been sharpened but not yet applied to a specific job. For years, the standard approach to solving a new problem with these tools was to combine as many of them as possible, operating under the assumption that more information always leads to better results. The logic was simple: if one tool sees a picture in one way and another sees it in a different way, putting them together should create a perfect, all-encompassing view. This idea, known as multi-view learning, has long been a cornerstone of how scientists try to build smarter systems by merging different sources of information.

However, a new study challenges this fundamental belief. Researchers from Beijing University of Posts and Telecommunications discovered that when you start mixing a large number of these pre-trained experts, adding more does not necessarily make the final result better. In fact, after a certain point, adding another tool can actually make the system worse. The team found that the best performance is usually achieved by selecting a small, specific group of these models rather than using a large, haphazard collection. They observed that while the first few models added to the mix provide fresh and useful insights, subsequent additions often repeat information the system already has or introduce confusing signals that have nothing to do with the task at hand. This phenomenon creates a curve where performance rises quickly, hits a peak, and then begins to fall as more models are forced into the mix.

To solve this problem, the researchers developed a new method called KAGES, which acts as a smart selector for these AI models. Instead of blindly fusing every available tool, KAGES carefully evaluates which specific models should be combined and how many should be used. It works by measuring how well the combined information from a group of models aligns with the correct answers for a specific task, without needing to train a new system to test them first. The method greedily adds the next best model that offers the most new, useful information while avoiding those that simply repeat what is already known or add noise. This approach allows the system to stop adding models exactly when the extra information stops being helpful, ensuring the final combination is compact and highly effective.

The team tested this idea across a wide variety of tasks, including recognizing objects in images, identifying textures, and even understanding language. In every case, they found that the "more is better" assumption failed. On some tasks, the performance peaked after just three or four models were combined, and adding more caused the accuracy to drop. For example, on a task involving recognizing traffic signs, the system performed best with a very small set of models, while on a task involving geolocation, a single model was already so good that adding others provided no benefit and sometimes hurt performance. The researchers also noted that the ideal number of models changes depending on how much data is available; with more data, the system can handle a few more models before the performance starts to decline, but the peak is always reached with a surprisingly small number.

The results showed that KAGES consistently outperformed other methods, including those that tried to combine all available models or those that simply picked models that were different from each other. By focusing on the quality of the combination rather than the quantity, KAGES was able to find the sweet spot where the models worked together most effectively. In many instances, it performed nearly as well as a perfect, theoretical selector that would have to test every possible combination to find the best one, but it did so much faster and without needing to train new systems for every test. This discovery suggests that the future of building powerful AI systems lies not in accumulating more tools, but in the careful, intelligent selection of the right ones. As the library of available AI models continues to grow, the ability to choose the perfect small team of experts will become more important than ever, turning the challenge of managing vast resources into an opportunity for precision and efficiency.

Technical Summary: When More Foundation Models Means Less

Problem Definition: View-Set Composition in Large Pools

Traditional multi-view learning operates under the assumption that a small, predefined set of complementary views (e.g., from different sensors or hand-crafted features) improves discriminative representation. However, the proliferation of pretrained foundation models has fundamentally expanded the candidate view space. In the frozen-encoder setting, each pretrained model induces a distinct representation view, creating a large, heterogeneous pool of encoders spanning various architectures, pretraining objectives, and modalities.

This shift transforms the design problem from simply fusing a fixed set of views to view-set composition: determining which encoders to combine from a large pool and how many to include. Empirical observations reveal that downstream performance does not improve monotonically with the number of fused encoders. Instead, performance typically peaks after a small subset is included and subsequently declines. This "peak-then-decline" phenomenon is driven by two failure modes:

Representation Redundancy: New encoders repeat information already captured by the current set.
Task-Model Misalignment: New encoders introduce task-irrelevant cues, where diversity alone does not equate to effectiveness.

Existing selection strategies are insufficient for this regime. Per-encoder transferability scores ignore interactions between encoders; diversity-based criteria prioritize difference over task utility; and exhaustive subset search is computationally infeasible for large pools. The paper posits that view-set composition is an ordering problem: a high-quality ordering should prioritize useful encoders so that its prefixes provide competitive fused sets under varying compute budgets.

Methodology: KAGES (Kernel-Alignment Greedy Encoder Selector)

To address the need for a training-free, set-level selection criterion, the authors propose KAGES. KAGES is a label-aware, greedy selector that constructs an encoder ordering without requiring downstream classifier training or combinatorial subset search.

Core Objective

KAGES defines the utility of a fused set $S$ based on the alignment between its representation kernel and the label kernel. Specifically, it utilizes Centered Kernel-Target Alignment (CKA).

Utility Function: $J(S) = U(\bar{F}_S, Y) - \tau \sum_{s \in S} \frac{d_s}{n}$ , where $U$ is the CKA between the joint representation kernel $K_S$ and the label kernel $K_Y$ . The second term is an optional complexity penalty based on encoder dimension $d_s$ .
Additivity: Under linear-kernel fusion (feature concatenation), the fused kernel is additive ( $K_S = \sum K_s$ ). This allows KAGES to evaluate marginal gains directly at the kernel level ( $O(n^2)$ per step) without retraining models for every candidate subset.

Selection Mechanism

KAGES operates greedily:

Initialize an empty set $S$ .
At each step, evaluate all candidate encoders $m \notin S$ by their marginal gain in set-level utility: $\Delta J(m | S) = J(S \cup \{m\}) - J(S)$ .
Select the encoder with the largest marginal gain and add it to $S$ .
Repeat until all candidates are ordered.

The marginal gain criterion admits a relevance-redundancy interpretation. The algorithm favors candidates that are well-aligned with the labels (relevance) while exhibiting limited kernel overlap with already-selected encoders (redundancy). Crucially, the redundancy weight is adaptive, determined by the current set's alignment and mass, rather than being fixed a priori.

Theoretical Guarantees

The paper provides a conditional theoretical guarantee. If the alignment utility is monotone and satisfies a submodularity ratio condition ( $\gamma$ ), the greedy prefix of size $k$ achieves a $(1 - e^{-\gamma})$ approximation to the optimal size- $k$ subset. Furthermore, the paper establishes a "single gain-cost crossing" mechanism (Theorem 1) explaining the order-dependent peak: as encoders are added, marginal gains diminish while finite-sample costs accumulate, eventually causing performance to decline.

Key Contributions

New Framing: Recasts large-pool frozen foundation-model fusion as multi-view learning at scale, formulating view-set composition as a central design problem.
Theoretical Characterization: Provides a marginal-utility account for the failure of the "more is better" assumption, identifying a gain-cost crossing that leads to order-dependent performance peaks and subsequent declines.
Scalable Selection Method (KAGES): Introduces a training-free, label-aware selector that evaluates candidates via marginal kernel alignment. It avoids repeated downstream fitting and offers conditional prefix-wise approximation guarantees.
Broad Empirical Validation: Validates the approach across image classification, image retrieval, and frozen large-language-model (LLM) fusion, demonstrating that the peak-then-decline phenomenon is structural and generalizable.

Experimental Results

The authors evaluated KAGES on five recognition regimes (generic object, fine-grained, texture, geo-spatial, structured symbol) using a pool of 10–15 frozen encoders spanning five pretraining paradigms (language-supervised, self-supervised, generative, class-supervised, task-expert).

Performance: KAGES consistently outperformed competing methods, including full fusion (Fuse-All), random ordering, diversity-based selection, and stronger subset-selection baselines (DPP, Submodular).
Metrics: Measured by AULC (Area Under the k-Accuracy Curve) and Acc@k* (peak accuracy). KAGES achieved the highest AULC across all protocols (16-shot, 8-shot with larger pool, and full data).
Specific Gains: On the structured-symbol task (GTSRB), KAGES improved over full fusion by +8.9 to +15.5 points. On full-data protocols, it led full fusion by an average of +3.3 points.
Behavior: KAGES successfully "front-loads" high-utility encoders (often self-supervised models like DINOv2 or class-supervised models like ResNet-50) and correctly rejects adversarial paradigms (generative and task-expert models) that do not align with the classification task.
Generalization: The peak-then-decline pattern and KAGES's superiority were observed in image retrieval and frozen-LLM fusion (MMLU and ARC-Challenge), confirming the method's applicability beyond standard image classification.

Significance and Claims

The paper argues that effective multi-view learning over foundation-model pools depends not on fusing more encoders, but on selecting a compact and complementary set of views. The authors claim that as public model hubs grow, the central question of multi-view learning shifts from "how to fuse a fixed few" to "which encoders to fuse."

KAGES provides a training-free, theory-backed answer to this question. By leveraging kernel alignment, it offers a scalable, set-level criterion that produces high-quality encoder orderings without the computational cost of training downstream predictors for every candidate subset. The work suggests that the "more is better" assumption is structurally flawed in the era of foundation models and that explicit view-set composition is necessary to unlock the potential of heterogeneous encoder pools.