🤖 machine learning

Human Preference aligned Tabular Similarity

This paper argues that current tabular embedding methods, optimized for prediction tasks, fail to align with human preferences for similarity search, and proposes a new evaluation procedure to address this gap using a Product Lifecycle Management (PLM) use case.

Original authors: Frederik Hoppe, Astrid Franz, Marianne Michaelis, Lars Kleinemeier, Udo Göbel

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Frederik Hoppe, Astrid Franz, Marianne Michaelis, Lars Kleinemeier, Udo Göbel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a massive, magical library where every book is actually a spreadsheet of numbers and words. In this library, there are robots whose job is to find books that are "similar" to the one you are holding. If you pick up a book about "fixing a broken toaster," the robot should hand you other books about toaster repairs, not books about baking cakes or driving cars. This is the world of tabular data—the organized grids of information that run our businesses, hospitals, and factories. The robots use something called embeddings, which are like invisible magnetic codes that turn each row of data into a point in a giant, multi-dimensional space. The closer two points are, the more similar the robot thinks the data is.

For a long time, scientists have been teaching these robots to get really good at guessing the answer to a specific question, like "Will this part break?" or "What is the price?" But here is the catch: being good at guessing an answer doesn't always mean being good at finding things that feel similar to a human. A robot might think two parts are twins because they both have the number "5" in them, even though a human engineer knows one is a screw and the other is a bolt. This paper asks a very important question: How do we make sure these robots are finding things that actually make sense to the different people using them, like engineers, factory workers, or shoppers?

The Problem: When Robots Get the "Vibe" Wrong

The authors of this paper, working with a company that builds software for managing product lifecycles, noticed a big gap. They saw that while robots are getting smarter at turning tables of data into those invisible magnetic codes, we don't have a good way to check if those codes match what humans actually care about.

Imagine you ask a robot to find "similar" change requests for a new car part. An engineer might say, "These are similar because they both deal with the engine's cooling system." A factory worker might say, "No, these are similar because they both require the same welding machine." A buyer might say, "These are similar because they come from the same supplier."

The paper argues that current ways of testing these robots are like grading a student only on their math test scores, even though the student is trying to become a poet. The standard tests measure if the robot can predict a number or a category correctly, but they don't ask: "Did the human user find the result useful?" The authors suggest that without asking humans what they think, we are flying blind. We might have a robot that is technically perfect at math but terrible at understanding human needs.

The Solution: A "Taste Test" for Data

To fix this, the authors propose a new workflow that sounds a bit like a blind taste test for food, but for data. Instead of just running the robot and hoping for the best, they suggest a three-step process to get human feedback:

The Setup: First, the robot creates its magnetic codes for all the data and stores them.
The Pick: A human user is shown a specific item (the "anchor"), like a specific part or a ticket. The robot then pulls up a list of items it thinks are the closest matches.
The Vote: The human looks at the original item and one of the robot's suggestions side-by-side. They then give it a rating: "Identical," "Similar," "Slightly Similar," or "Not Similar."

This isn't just a one-time thing. The authors suggest doing this over and over, collecting thousands of these "votes." This creates a map of what different groups of people actually consider similar.

What They Found: Everyone Sees the World Differently

To test their idea, the team ran a small pilot study. They took a list of 20 different "tickets" (like requests for help or changes) and asked three different people to rate the top 6 matches the robot found for each one.

The results were eye-opening. Out of 120 pairs of items they looked at, the three people only agreed on the rating for 63 pairs (which is 52.5%). In 52 pairs (43.3%), two people agreed but the third disagreed. And in 5 pairs (4.2%), all three people gave different ratings!

This proves that "similarity" is not a single, fixed fact. It depends entirely on who you are. An engineer might see a pattern that a purchaser misses, and vice versa.

The team then tested 10 different robot algorithms to see which one was the best at guessing what these humans wanted. Here is the twist: There was no single "winner."

For Annotator 1 and Annotator 3, Algorithm 7 was the best.
But for Annotator 2, Algorithm 4 was the best.

Even more interesting, the difference between the "best" and "second best" for Annotator 2 was tiny—just one single triplet (a group of three items) out of 165. This suggests that with such a small number of votes, it's hard to be 100% sure which robot is truly better. The authors point out that we need more data to be confident, but the main takeaway is clear: One size does not fit all. A robot that is perfect for an engineer might be useless for a buyer.

Why This Matters for Trust

The paper concludes that for AI to be truly trustworthy, we can't just rely on the robot's internal math. We need to build a system where humans can constantly check the robot's work.

They highlight four key benefits of this approach:

Fairness: It ensures the robot treats different groups of people (like engineers vs. buyers) fairly, rather than favoring one perspective.
Transparency: It lets people see why the robot picked a certain item, making the "black box" a little less dark.
Robustness: It helps find messy data, like duplicates or confusing categories, that the robot might be getting wrong.
Accountability: It creates a paper trail of human decisions, which is crucial if the AI makes a mistake in a regulated industry.

The authors admit that asking humans to do this is hard work and takes time. They don't claim to have solved the problem perfectly yet. Instead, they are calling on the scientific community to start using these "human taste tests" as a standard part of building AI. They argue that trust isn't something you bake into the code once; it's something you build by constantly listening to the people who use the system.

Technical Summary: Human Preference Aligned Tabular Similarity

Problem Statement

While task-agnostic tabular embeddings are increasingly utilized for similarity search in industrial systems like Product Lifecycle Management (PLM), a critical gap exists in their evaluation. Current leading embedding approaches are optimized for prediction tasks (classification, regression) rather than producing similarity rankings that align with human preference. Standard downstream metrics (e.g., accuracy, F1-score) are insufficient for assessing trustworthiness in similarity search because they fail to capture "human-perceived similarity."

In industrial contexts, similarity is highly subjective and user-dependent. For instance, in a PLM system, an engineer may define similarity based on geometric properties, a production worker by manufacturing processes, and a purchaser by supplier origin. Consequently, an embedding model that performs well on objective classification benchmarks may retrieve nearest neighbors that are structurally similar but semantically meaningless to specific domain experts. Furthermore, existing Information Retrieval (IR) metrics (e.g., MRR, NDCG) and Reinforcement Learning from Human Feedback (RLHF) approaches do not adequately address the heterogeneity of tabular data or the diverse, often conflicting, preferences of different user groups.

Methodology

The authors propose a concrete evaluation workflow designed to capture human preference alignment for tabular similarity. This workflow consists of three primary steps:

Ingestion: Tabular records are embedded using a pre-selected model and indexed in a vector store. Configuration details, including the embedding model and distance function (e.g., cosine distance), are stored as metadata.
Anchor Selection: An anchor record is selected from the dataset. Selection strategies include random sampling (potentially guided by uncertainty or diversity sampling), attribute-based filtering, hybrid retrieval, or direct user selection based on business criticality. The system retrieves the nearest neighbors for this anchor based on the predefined distance function.
User Assessment: The anchor and a neighboring record are displayed side-by-side. Users assign a pairwise similarity label (e.g., "identical," "similar," "slightly similar," "not similar"). These labels are stored alongside anonymized user and session metadata. To ensure data quality, inter-annotator agreement metrics (e.g., Cohen's $\kappa$ ) are monitored, and disagreements are flagged for analysis.

The authors note that while Approximate Nearest Neighbor (ANN) search is often used for scalability, exact nearest neighbor search or tuned ANN hyperparameters should be considered during annotation to minimize retrieval noise.

Key Contributions

Identification of the Evaluation Gap: The paper argues that objective downstream metrics are inadequate for assessing the trustworthiness of tabular embeddings in similarity search, specifically highlighting the lack of human preference alignment.
Proposed Evaluation Workflow: A structured, three-step procedure for collecting human preference rankings is introduced. This workflow facilitates active learning and allows for the automated quantitative evaluation of embedding algorithms based on user-specific relevance.
User-Group Specificity: The framework explicitly supports the generation of user-group-specific rankings. It acknowledges that an optimal embedding configuration for one group (e.g., engineers) may differ from another (e.g., purchasers), challenging the notion of a single "best" global model.
Integration with Trustworthy AI: The authors map their proposed workflow to four pillars of Trustworthy AI:
- Fairness: Enabling equitable treatment of different user groups.
- Transparency: Making neighborhood structures and retrieval logic visible to users.
- Robustness: Using similarity labels to identify data quality issues (duplicates, inconsistencies).
- Accountability: Providing persistent records for governance and quality assurance.

Results: Pilot Study

The authors conducted a pilot study using a PLM ticket dataset containing various categories (IT, safety, permission).

Setup: Three annotators rated the top-6 nearest neighbors for 20 anchor tickets using a 4-point scale, generating 120 ticket pairs with three annotations each.
Divergence: Significant disagreement was observed. Only 52.5% of pairs received identical ratings across all three annotators, while 43.3% were rated in two different categories and 4.2% in three. This suggests that disagreement often reflects legitimate, domain-specific differences in perspective rather than random noise.
Algorithm Evaluation: The study evaluated 10 different embedding algorithms using triplet accuracy (measuring if the distance to a "more similar" item is smaller than to a "less similar" item).
- Performance Variance: Algorithm 7 performed best for Annotators 1 and 3, while Algorithm 4 was superior for Annotator 2.
- Statistical Sensitivity: The performance difference between the top algorithms for Annotator 2 (Algorithms 4 and 7) was marginal (a difference of a single triplet). This highlights the need for sufficient annotation volume to achieve statistical confidence in identifying group-specific preferences.

Significance and Claims

The paper positions itself as a call to action for the community to adopt broader evaluation standards aligned with Trustworthy AI principles. The authors do not claim to have solved the problem of tabular similarity but rather outline the necessity of human-centered validation.

Emergent Trustworthiness: The paper argues that the trustworthiness of AI systems is not an intrinsic property of the embedding model itself but an emergent property of robust socio-technical processes. These processes must continuously inspect, challenge, and document model behavior.
Lifecycle Integration: User-group-specific embedding validation must be integrated as a fundamental component of the AI lifecycle and risk management frameworks.
Future Directions: The authors identify four critical questions for the community to address:
1. How much annotation is sufficient to represent varied user perspectives?
2. How should legitimate disagreements between annotators (stemming from different user groups) be handled?
3. How can bias in embedding neighborhoods be detected and mitigated across different user groups?
4. How can preference ranking results be fed back into system improvement to accommodate diverse needs?

The work emphasizes that while resource-intensive, human preference alignment is necessary to ensure that similarity search systems perform equitably and satisfy the diverse relevance criteria of different stakeholders in industrial applications.