📊 statistics

Fair Multi-View Determinantal Coresets via Adaptive NEPv

This paper introduces a fair multi-view determinantal coreset selection method that maximizes the weakest per-view log determinant by formulating a gauge-invariant nonlinear eigenvalue problem, which is solved via an adaptive self-consistent-field algorithm and rounded using leverage-score screening with local refinement.

Original authors: Richard Yi Da Xu

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Richard Yi Da Xu

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, building a smart system often begins with a curation problem: a massive library of data exists, but the computer can only learn from a tiny fraction of it. The challenge is not just picking the best examples, but picking the most useful variety. Imagine trying to teach a machine to recognize a brand by showing it a few logos and their written descriptions. If you only choose examples that look different from one another, you might accidentally pick a set where every logo is unique but every description uses the exact same boring words. Conversely, if you only pick examples with diverse wording, you might end up with a set where the text is rich but the images are all nearly identical. This creates a blind spot. The machine learns to handle one side of the story while failing completely on the other. This is the core difficulty of "multi-view" learning, where data comes in different forms, such as text and images, and a good selection must satisfy the requirements of every form simultaneously.

Researchers at Hong Kong Baptist University and TadReamk Limited have developed a new method to solve this specific balancing act. They call their approach "Fair Multi-View Determinantal Coresets." The goal is simple in concept but difficult to achieve: select a small group of items that is diverse in every single way it can be measured, without letting one type of diversity hide the failure of another. To do this, they moved away from the old habit of averaging different types of data into a single score. Averaging is deceptive because a high total score can hide the fact that one part of the data is completely collapsed. Instead, their new method focuses on the weakest link. It asks, "What is the least diverse view in this group?" and then tries to make that specific view as diverse as possible. By constantly pushing up the floor of the worst-performing category, the method aims to ensure that no single perspective is neglected, though it does not claim to guarantee a perfectly balanced selection in all cases.

The mathematical engine behind this solution is a sophisticated way of handling the selection process. Normally, picking a subset of items is a discrete choice, like flipping a switch for each item to say "yes" or "no." This is computationally very hard when the list of candidates is huge. The researchers transformed this problem into a continuous one, where they imagine the selection as a shape that can rotate and shift smoothly in a high-dimensional space. This allows them to use powerful mathematical tools to find the best orientation for this shape. However, because they are trying to balance multiple different views at once, the rules for finding the best shape change as the shape itself moves. It is not a static calculation; the weights assigned to the different views adapt based on how well the current selection is performing in each area. If the text diversity is low, the system automatically puts more pressure on the text side to improve.

To solve this moving target, the team built a custom solver that iterates toward a solution. It starts with a random selection and repeatedly adjusts the group, checking which view is lagging behind and shifting the focus to fix it. They added specific techniques to keep this process stable, preventing it from oscillating wildly or getting stuck. Once the system finds the best possible continuous shape, it translates that back into a concrete list of actual items. This final step involves a screening process to pick the top candidates, followed by a local refinement where items are swapped in and out to ensure the final list is as good as possible. While the method seeks a balanced selection, the author notes that the mathematical relaxation used to solve the problem does not always perfectly match the discrete reality, meaning the final result may not be a perfect guarantee of balance.

The researchers tested their method using synthetic data designed specifically to create conflict. They created a scenario with three types of candidates: some that were excellent at text but poor at images, some that were excellent at images but poor at text, and some that were mediocre at both. When they used traditional methods that averaged the scores or looked at only one view, the system picked a group that was heavily biased toward one side, leaving the other side with almost no variety. The new fair method, however, successfully identified a mix that included candidates from both sides, ensuring that both the text and the image aspects of the group were diverse. In these controlled simulations, the new method achieved a significantly higher score for the weakest view compared to all other approaches, proving that it could effectively balance conflicting requirements.

While the method has been proven to work in these simulated environments, the author is careful to note that they have not yet tested it on real-world data in this specific report. They have outlined a detailed plan to apply this technique to a massive database of United States trademark records, which contain both logo images and the legal text descriptions filed by applicants. This real-world test would involve training a large language model to generate descriptions based on the selected logos. The researchers intend to release the specific data and code needed to run this test, but the actual results from the trademark data are not included in this version of the work. For now, the contribution is a robust mathematical framework and a solver that seeks a balanced selection in theory and in simulation, offering a new way to ensure that AI training data is truly representative of all the different ways information can be presented, without claiming a guaranteed approximation for the final discrete set.

Technical Summary: Fair Multi-View Determinantal Coresets via Adaptive NEPv

Problem Statement

In supervised multimodal learning, curation under a fixed budget often requires selecting a diverse subset of candidates that carry multiple representations (views), such as text descriptions and logo images. A critical bottleneck arises when a single kernel or an averaged kernel fails to capture diversity across all views simultaneously. For instance, a subset selected for textual variety may collapse in visual diversity, and vice versa. Standard approaches, such as selecting under a single view or averaging kernels, either ignore specific views or mask failures in one view by a high aggregate score.

The paper addresses Fair Multi-View Determinantal Selection: the problem of selecting a size- $k$ subset $S$ that maximizes the weakest per-view normalized log-determinant. Formally, given $M$ views with kernels $L^{(1)}, \dots, L^{(M)}$ , the goal is to maximize:
$\min_{q} \left( \frac{1}{k} \log \det(L^{(q)}_S) \right)$
This objective ensures that no single view is sacrificed for the sake of others, preventing "blind spots" where one modality is under-covered.

Methodology

The author proposes a continuous relaxation of this discrete problem using the Stiefel manifold and solve it via an Adaptive Nonlinear Eigenvalue Problem with Eigenvector Dependency (NEPv).

Stiefel Relaxation:
Instead of using fractional membership weights (simplex relaxation), the method relaxes the discrete indicator matrix $V_S \in \{0,1\}^{n \times k}$ to an orthonormal frame $V \in \text{St}(n, k)$ (where $V^\top V = I_k$ ).
- This preserves the exact geometry of $k$ -dimensional volumes for each view.
- Crucially, it evaluates each original log-determinant separately rather than replacing them with a single averaged kernel.
- The objective function is smoothed using a soft-min approximation (finite temperature $\tau$ ) to handle the non-differentiable $\min$ operator:
  $f_\tau(V) = -\tau \log \left[ \frac{1}{M} \sum_{q=1}^M \exp\left( -\frac{g_q(V)}{\tau} \right) \right]$
  where $g_q(V) = \frac{1}{k} \log \det(V^\top L^{(q)} V)$ .
Adaptive NEPv Formulation:
Unlike single-view DPP-MAP, which has a closed-form spectral solution (top- $k$ eigenvectors), the multi-view fair objective yields stationarity conditions that depend on the current subspace.
- The gradient of the objective leads to view-adaptive weights $w_q(V)$ , which assign higher weight to views with lower log-determinants (the "weakest" views).
- The stationarity condition is derived as a gauge-invariant NEPv:
  $H_\tau(V) V = V \Lambda$
  where $H_\tau(V)$ is a symmetric operator constructed from the weighted sum of view-specific inverse Gram matrices. The operator depends on $V$ itself, making the problem nonlinear.
Solver (FairNEPv):
The author develops an adaptive Self-Consistent Field (SCF) iteration to solve the NEPv:
- Iteration: At each step, compute view-specific Gram matrices and weights, construct the operator $H_\tau(V_t)$ , and extract the top- $k$ eigenvectors to update $V_{t+1}$ .
- Stabilization: The solver employs damping (to handle oscillations when the "worst" view changes) and level shifting (to ensure a uniform eigengap lower bound, guaranteeing the top- $k$ subspace is well-defined and unique).
- Rounding: The continuous subspace $V^*$ is rounded to a discrete set $S$ via leverage-score screening followed by a local greedy refinement (one-item swaps) to maximize the discrete fair objective.

Key Contributions

Fair Multi-View Stiefel Relaxation: The paper formulates a relaxation that retains separate determinants for each view and maximizes a smooth approximation of the minimum. It proves that discrete subsets embed exactly into this relaxation, though the resulting stationarity equations differ from fixed-weight kernel problems.
Adaptive NEPv and SCF Solver: The author derives a gauge-invariant NEPv where the operator depends on the subspace via soft-min weights. They provide a local convergence theorem for the idealized subspace map, showing contraction under a specific sensitivity condition relative to the eigengap.
Reproducible Multimodal Protocol: The paper specifies a protocol for the USPTO trademark dataset, aligning text descriptions with logo embeddings. It outlines a rigorous evaluation framework including intrinsic fairness metrics; however, the author explicitly notes that real-data multimodal results and downstream evaluation are not included in this version pending the release of aligned image features and splits.

Results

Synthetic Experiments: In a controlled "quality-conflict" stress test (where candidates are either text-specialists, vision-specialists, or balanced), FairNEPv successfully selects a balanced mix of items.
- Single-view selectors collapsed one view entirely.
- Averaged kernels and fixed-weight sums failed to enforce balance, often selecting only specialists from one side.
- FairNEPv achieved a fair score ( $F_\tau$ ) of 1.120, significantly outperforming baselines (e.g., $-2.199$ for single-view selectors). The gap to the discrete optimum (1.141) was small (0.021).
Real-World Data: The paper does not report results on the USPTO dataset. It explicitly states that the real-data multimodal results are omitted because the aligned logo embeddings and image features are not yet released in the repository. The empirical study is specified but deferred until the necessary data artifacts are available.

Significance and Claims

The paper modestly claims to provide a theoretical framework and algorithmic solver for fair multi-view curation.

It establishes that the Stiefel relaxation allows for exact embedding of discrete subsets while enabling continuous optimization.
It demonstrates that the multi-view fairness problem inherently leads to a nonlinear eigenproblem, necessitating the proposed adaptive SCF approach rather than standard spectral methods.
The author explicitly states that their work does not claim global convergence, a rounding approximation guarantee, or proven downstream utility gains on real data. The significance lies in isolating the fairness mechanism in synthetic settings and defining a protocol for future multimodal evaluation. The real-data validation is deferred until the necessary data artifacts are released.