🤖 AI

Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

This paper introduces a comprehensive diagnostic testbed and method comparison for selective question answering over conflicting multi-source personal memory, demonstrating that trained fusion resolvers outperform frontier LLMs in accuracy and selective abstention capabilities across a large-scale, controlled benchmark.

Original authors: Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Confused Detective" Problem

Imagine you are a detective trying to solve a mystery about someone's daily life. You have five different witnesses giving you testimony:

The Long-Term Profile: An old biography written years ago (might be outdated).
The Planner: A diary of what the person intended to do (often overly optimistic).
The Self-Report: A daily journal written by the person (they might lie to look good or forget details).
The Objective Log: A receipt book or timestamp record (accurate, but might have missing pages).
The Device Log: A smartwatch record (very precise, but the battery might die, leaving gaps).

The Problem: These witnesses often disagree. The planner says, "I ran 5 miles!" The self-report says, "I ran 5 miles!" But the device log says, "0 miles," and the objective log is silent.

Current AI agents often just pick one witness or try to guess the answer blindly. This paper asks: How do we build an AI that knows which witness to trust, and when to admit, "I don't know, the evidence is too messy"?

The Solution: A "Training Gym" for AI

The authors built a massive diagnostic testbed (a practice gym) to train and test AI on this specific problem. They didn't use real people's private data (which would be a privacy nightmare); instead, they created 480 fake "personas" (digital characters) with known, secret truths about their lives.

They generated 34,560 scenarios where these digital characters' five witnesses gave conflicting stories. The "Ground Truth" (the actual answer) was known to the researchers but hidden from the AI.

The Goal: The AI had to answer questions like, "How many days did this person sleep less than 6 hours?" or "Did they actually work overtime?" based only on the conflicting witness statements.

The Experiment: Who Wins the Detective Contest?

The researchers tested different types of "detectives" (AI methods) to see who could solve the mystery best.

1. The "Gut Feeling" Detectives (Baseline LLMs)

These are powerful AI models (like GPT-5.4 or Gemini) that read all the witness statements and just try to guess the answer.

Result: They were okay, getting about 70% of the answers right.
The Flaw: When the witnesses disagreed, the AI often got confused or trusted the wrong witness (like trusting the optimistic planner over the accurate device log).

2. The "Statistical Analysts" (Structured Fusion Methods)

These methods don't just read the text; they first extract the raw facts (e.g., "Planner says 5 miles," "Device says 0 miles") and then use a mathematical formula to weigh the reliability of each witness based on past training.

Result: The best one, called DSNBF, got 80.3% of the answers right.
Why they won: They learned that "Self-Reports" tend to exaggerate exercise, while "Planners" are usually too hopeful. They built a mental map of how each witness tends to lie.

3. The "Honest Detectives" (Selective QA)

This is the most important part. Sometimes, the evidence is so conflicting that no one can know the answer. A good detective should say, "I'm not sure," rather than guessing and being wrong.

The Result: When the best statistical method was allowed to say "I don't know" (abstain) on the hardest cases, its accuracy on the cases it did answer jumped to 85.3%.
The LLMs' Struggle: The standard AI models rarely said "I don't know." They kept guessing, which lowered their overall reliability.

Key Takeaways from the Paper

1. "Reading" isn't enough; "Reasoning" is key.
Simply asking a smart AI to read the conflicting stories and pick an answer isn't enough. The AI needs a separate "brain" (a resolver) that has been trained to understand how each source of information is biased. It's like knowing that your friend who loves sports always exaggerates their workout, so you mentally adjust their story before believing it.

2. Knowing when to stop is a superpower.
The best systems didn't just answer more questions; they knew when not to answer. By skipping the 20% of cases where the evidence was too messy, they became much more accurate on the remaining 80%.

3. The "Black Box" problem.
When you just ask an AI to "read and answer," you can't tell if it failed because it misread the text or because it couldn't resolve the conflict. This paper separates the two: first, extract the facts; second, resolve the conflict. This helps us see exactly where the AI is breaking.

What This Means (According to the Paper)

The paper concludes that for personal AI agents (like a digital assistant that knows your schedule, health, and habits) to be reliable, they cannot just be "chatbots." They need a structured layer that:

Extracts facts from different sources.
Learns which sources are trustworthy for specific topics.
Has the confidence to say "I don't know" when the sources are too contradictory.

The authors released their "gym" (the data and code) so other researchers can test their own AI detectives to see if they can solve these conflicts better. They emphasize that this is a diagnostic tool to find weaknesses, not a finished product ready for real-world deployment yet.

Technical Summary: Selective QA over Conflicting Multi-Source Personal Memory

1. Problem Definition

Emerging personal AI agents are transitioning toward persistent, multi-source memory architectures that aggregate data from long-term profiles, recent logs, plans, self-reports, and device records. A critical evaluation gap exists: these systems must resolve conflicting or incomplete evidence from heterogeneous sources rather than retrieving facts from a single, clean history. Existing benchmarks often fail to distinguish whether errors stem from the initial evidence extraction or the subsequent conflict-resolution step. Furthermore, current systems rarely support "selective QA," where an agent can abstain from answering when evidence is insufficient or irreconcilably conflicting, a capability essential for reliable deployment.

The paper formalizes this challenge as Selective QA over Conflicting Multi-Source Personal Memory. The core task requires a system to answer closed-class questions based on five biased, potentially conflicting evidence streams or to abstain (SKIP) when the evidence does not support a confident answer.

2. Methodology and Benchmark Design

The Diagnostic Testbed

The authors introduce a synthetic diagnostic testbed designed to isolate specific failure modes in conflict resolution:

Scale and Structure: The dataset comprises 34,560 instances generated from 480 personas across 4 random seeds. It includes 18 question templates covering 8 reasoning types (e.g., Source Arbitration, Plan vs. Reality, Missing Evidence) across five domains (Work, Diet, Social, Sleep, Exercise).
Data-Generating Process (DGP): The DGP operates as a Directed Acyclic Graph (DAG) to ensure deterministic ground truth (GT):
1. Latent Events (L2): A 30-day log of actual user behavior is generated.
2. Source Projection (L3): The latent events are projected into five distinct source streams (Profile, Planner, Self-Report, Objective Log, Device Log), each with specific, learnable distortion models (e.g., planners are optimistic, self-reports are socially desirable, device logs have missingness).
3. Natural Language (NL) Render: Structured projections are rendered into natural language memory documents.
4. Extraction: An LLM (GPT-5.4) extracts structured "atoms" (closed-class values) from the NL text.
Ground Truth: Labels are computed deterministically from the latent event table and specific source annotations, ensuring no circular dependency on the extraction process.
Evaluation Protocol: The benchmark separates extraction (converting NL to atoms) from resolution (mapping atoms to answers). This allows for a factorial decomposition of performance between the input quality (extracted atoms vs. direct structured readout) and the resolver logic.

Evaluated Methods

The study evaluates four tiers of methods:

T0 (Source-Free): Random and Majority Class baselines.
T1 (Source Selection): Selecting the single best source per question (SSB).
T2 (Structured Fusion): Methods that learn source-to-GT mappings:
- Majority Vote / ArgRAG-style: Simple aggregation.
- Bias Corrected Fusion (BCF): Applies hand-specified ordinal bias priors.
- Naive Bayes Fusion (NBF): Learns per-source confusion matrices from training labels.
- Difficulty-Stratified NBF (DSNBF): Learns confusion matrices stratified by persona difficulty classes, inferring the difficulty of test personas from their atom patterns.
- Abductive Bias Fusion (ABF): Uses a soft-mixture kernel blending bias-aware and bias-free paths.
T3 (End-to-End LLMs): Frontier models (GPT-5.4, Gemini 3.1 Pro, DeepSeek V3.2, Qwen3-235B) reading raw NL memory directly, with and without "Schema-Aware" prompts that inject qualitative bias descriptions.
Reference: Source Reachability (a GT-aided check of whether GT is directly visible in any source).

All methods support Selective QA variants that abstain (SKIP) based on calibrated thresholds (for fusion methods) or self-reported flags (for LLMs).

3. Key Results

Accuracy and Resolution

Fusion vs. Prompting: The best trained fusion resolver (DSNBF) achieves 80.3% macro accuracy on extracted atoms, outperforming the strongest prompt-only LLM baseline (GPT-5.4 Schema-Aware) by 10.3 percentage points (70.0%).
Input Quality: When provided with direct structured readout atoms ( $\mu^*$ ) rather than extracted atoms ( $\hat{\mu}$ ), DSNBF improves to 82.3%, while GPT-5.4 improves to 73.5%. The gap remains significant (8.8 pp), indicating that the resolver's ability to model source reliability is the primary driver of performance, not just input extraction quality.
Source Selection: Naive aggregation (Majority Vote) performs poorly (~69%) because biased sources can outvote accurate ones. Single-source selection (SSB) performs better (77.3%) but is outperformed by learned fusion methods.

Selective QA (Abstention)

Fusion Methods: DSNBF with selective abstention achieves 85.3% selective accuracy at 78.3% coverage. This indicates the method successfully identifies and skips difficult cases where evidence is insufficient.
LLMs: Frontier LLMs show limited ability to abstain effectively. GPT-5.4 reaches 71.0% selective accuracy at 95.4% coverage, and other models (Gemini, DeepSeek, Qwen) abstain on fewer than 2% of instances, failing to leverage abstention to improve accuracy.

Diagnostic Insights

Reasoning Types: DSNBF leads on 7 of 8 reasoning types. The largest gaps appear in Arbitration (+19.0 pp) and Factor Attribution (+12.5 pp), where learning per-source confusion matrices is critical.
Difficulty Classes: Performance degrades on "stated_vs_revealed" personas (where stated preferences contradict revealed behavior). Fusion methods show robustness (DSNBF drops only ~~5 pp), while schema-prompted LLMs suffer significant degradation (~~13 pp), suggesting qualitative bias hints are insufficient for complex conflict resolution.
Robustness: Results are stable across random seeds, training sizes (saturating at ~100 personas), and DGP perturbations (bias and dropout scales).

4. Contributions

Task Formalization: Defines bias-aware selective QA over multi-source personal memory, explicitly modeling directional source distortions and the option to abstain.
Diagnostic Testbed: Releases a synthetic benchmark (34,560 instances) with a documented DGP, separating extraction from resolution to enable controlled analysis of failure modes.
Diagnostic Evaluation: Demonstrates that learned source models (confusion matrices) and calibrated abstention significantly outperform prompt-only LLMs and naive aggregation, revealing that conflict resolution is a distinct capability from information extraction.

5. Significance and Limitations

The paper argues that for personal AI agents to be reliable, conflict handling and the ability to abstain must be made explicit before deployment. The separation of extraction and resolution allows developers to localize failures: whether a system fails because it cannot read the memory, cannot extract facts, or cannot weigh conflicting evidence.

Limitations:

Synthetic Nature: The data is synthetic with monotone, topic-dependent distortions. Real-world data may involve non-monotonic habit changes, adversarial reports, or complex sensor failures not captured here.
Template Count: The 18 question templates limit statistical power within specific reasoning types, though the DGP supports expansion.
Training Requirement: The strongest methods (DSNBF, NBF) require labeled training personas. The authors caution that these results should not be interpreted as claims for cheaper, zero-shot deployment but rather as a diagnostic baseline for what is achievable with supervision.

The work concludes that while frontier LLMs can extract facts, they currently struggle with the specific task of resolving systematic, multi-source conflicts without explicit training on source reliability patterns. The released code and data aim to facilitate further research into reliable personal memory agents.