🤖 AI

Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfaces

This paper proposes an Inverse Theory of Mind (IToM) pipeline that uses LLM-driven counterfactual reasoning and abductive inference to reconstruct user beliefs and generate structured personas from browsing interactions, demonstrating superior accuracy in predicting user traits and enabling dynamic, modality-agnostic content recommendations across applications like VisionOS.

Original authors: Mengyu Chen, Feiyu Lu, Chun-Fu Chen, Lucas Vinh Tran, Jay Katukuri

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mengyu Chen, Feiyu Lu, Chun-Fu Chen, Lucas Vinh Tran, Jay Katukuri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what your friend wants for dinner. If you only watch them click on a pizza ad once, you might think, "Aha! They love pizza!" But what if they were just comparing prices? Or maybe they were looking for a gift for someone else? In the world of computers, this is the big problem with how we usually build "recommender systems." These are the smart algorithms that suggest movies, songs, or products. Traditionally, they act like a statistician who only counts how many times you click a button. They see the action (the click) but they don't understand the why behind it. They don't know if you clicked because you love the item, because you were bored, or because you were confused.

To fix this, scientists have been looking at a concept from psychology called "Theory of Mind." Think of this as the ability to imagine what is going on inside someone else's head—their beliefs, their goals, and their secrets. Usually, computers try to use this to guess what you will do next based on what they think you want. But this paper flips the script. It asks: What if we could work backward? What if, instead of guessing the future, we could look at your past clicks and use a kind of "reverse detective work" to figure out who you actually are? This is the core idea: taking messy, everyday digital footprints and using them to build a rich, understandable story about a person's personality, without ever having to ask them a single survey question.

The researchers behind this study, working at JPMorgan Chase, propose a new method called "Inverse Theory of Mind" (IToM). Think of it as a digital detective agency. Instead of just recording that you clicked on a pair of running shoes, the system asks, "Okay, what was on the screen when you clicked? What other shoes were there? Why did you pick these ones and not the others?" By reconstructing the exact moment of decision, the system uses a powerful AI (a Large Language Model) to reason backward. It generates a list of "beliefs" that explain your behavior, such as, "This person values durability over style," or "They are a careful researcher who reads reviews before buying."

Here is where the magic happens: the system doesn't just pick one guess. It knows that human behavior is tricky. Maybe you bought the shoes because you love running, or maybe you bought them because they were on sale and you needed a gift. To handle this uncertainty, the AI creates multiple different versions of your personality profile, like a team of detectives each offering a different theory. Then, it carefully weighs all these theories together. If the evidence is strong, it leans into the specific theory; if the evidence is weak, it plays it safe and leans toward what is generally true for most people. This prevents the AI from making wild, stereotypical guesses (like assuming every careful shopper is an anxious introvert).

The team tested this idea using a special dataset of real Amazon shopping sessions, which included not just the clicks, but also the actual pages people saw and, crucially, real interviews and personality tests that the shoppers had taken. They found that their "reverse detective" personas were surprisingly good. In fact, in some tests, the AI's guess about a user's personality was just as good as, or even better than, the actual personality test the user had taken. They showed that these AI-generated profiles could predict what a user would do next, what kind of products they would like, and even how they would answer questions about their shopping habits.

Perhaps the most exciting part is that these profiles aren't just numbers hidden inside a computer. They are written in plain, human language. This means the computer can take a profile built from 2D web browsing and use it to design a 3D interface for a futuristic headset (like the Apple Vision Pro). For example, if the AI figures out you are a "budget-conscious planner," it can automatically arrange a 3D banking app to show your savings goals front and center, while hiding flashy investment offers. The paper suggests that this approach bridges the gap between simple click-tracking and deep human understanding, offering a way to build smarter, more empathetic interfaces that adapt to who we are, not just what we click. However, the authors are careful to note that while the results are promising, the system still has limits; it works best with large amounts of data and sometimes struggles to predict extreme personality traits perfectly, suggesting that while the "reverse detective" is getting good at its job, it still has room to learn.

Technical Summary: Inverse Theory of Mind Modeling for Content Recommendation

Problem Statement

Modern recommender systems predominantly rely on observed interaction histories (clicks, views, scrolls) to predict future behavior. While effective for next-item prediction within a specific interaction type, these systems suffer from three critical limitations:

Scope-Bound Prediction: They learn statistical patterns of specific behaviors (e.g., clicks) but cannot infer the underlying rationale, goals, or personality traits driving those actions.
Inability to Adapt to New Modalities: As interfaces evolve toward generative UIs and immersive Extended Reality (XR), systems must decide not just what to show, but where, when, and how to present it. Interaction histories alone lack the rich user context required for these decisions.
Data Inefficiency of Persona-Based Methods: Traditional persona-based approaches offer rich user models (interests, values, personality) but rely on costly, intrusive qualitative data (interviews, surveys) that do not scale.

The core problem is the lack of a mechanism to bridge data-rich interaction traces with rich, modality-agnostic user understanding without requiring explicit qualitative data collection.

Methodology: The Inverse Theory of Mind (IToM) Pipeline

The authors propose an Inverse Theory of Mind (IToM) pipeline that reasons backward from observed actions to infer the beliefs, preferences, and decision-making traits that explain them. This approach treats user modeling as an abductive inference problem: inferring the best explanation (beliefs/persona) for observed evidence (actions).

The pipeline consists of five sequential stages, utilizing Large Language Models (LLMs) as approximate inverse reasoners:

Action Summarization: Raw interaction events (clicks, inputs, scrolls) are converted into concise, natural language sentences. Complex actions incorporate product details via LLMs, while simple actions use deterministic rules.
Percept Reconstruction: To address the lack of decision context in standard logs, the system reconstructs the "percept" for each action. Using the action summary and page HTML, the LLM identifies:
- Main choices: The primary decision set.
- Sub choices: Secondary alternatives.
- Other choices: Irrelevant options.
- User choice: The actual action taken.
  This reconstruction establishes the "consideration set" necessary for inverse reasoning.
Backward Belief Inference: The system performs abductive inference at two levels:
- Action-Level: Infers micro-signals (specific preferences, fine-grained traits) for individual actions by comparing the chosen option against rejected alternatives.
- Session-Level: Aggregates action-level hypotheses to identify macro-patterns (lifestyle signals, decision-making styles) across a session.
- Constraint: Every belief must be grounded in the reconstructed percept and cite specific chosen/rejected options to prevent hallucination.
Belief Optimization: The initial candidate set is often redundant. The system applies Maximal Marginal Relevance (MMR) to select a compact, diverse subset of beliefs, prioritizing rare and distinctive inferences over common shopping paraphrases.
Multi-Hypothesis Persona Synthesis: Recognizing that behavioral evidence is under-determined (multiple personality types could explain the same behavior), the system generates $M$ $M$ diverse persona hypotheses.
- Each hypothesis includes a profile (interests, values), a shopping profile, and evidence confidence levels.
- Final predictions (e.g., personality scores) are derived via confidence-weighted aggregation. When evidence is weak, the system adapts toward population priors; when strong, it relies on the hypotheses. This mechanism is designed to mitigate "persona bias," where single interpretations default to stereotypes.

Key Contributions

IToM Paradigm: The paper introduces Inverse Theory of Mind as a bridge between interaction-based recommenders and persona-based methods, instantiating it as a multi-stage abductive inference pipeline that constructs personas solely from behavioral traces.
Multi-Hypothesis Persona Synthesis: A novel approach that generates diverse personality interpretations of the same evidence and aggregates them with adaptive confidence weighting, rather than committing to a single potentially stereotypical interpretation.
Validation on OPeRA: The approach is validated on the OPeRA dataset, which uniquely pairs fine-grained Amazon browsing logs with ground-truth interview transcripts, personality assessments, and shopping attitude surveys.
Cross-Modal Transferability: A proof-of-concept demonstration on VisionOS, showing that personas inferred from 2D web browsing data can drive 3D spatial interface layouts and content selection without retraining.

Experimental Results

The pipeline was evaluated on 46 users from the OPeRA dataset across four tasks:

Next Action Prediction: Inferred personas outperformed ground-truth (GT) personas assembled from interviews on the Claude Opus 4.6 model, achieving a 50.6% relative improvement in action generation accuracy (25.69% vs. 17.06%). This suggests that grounding personas in specific behavioral episodes reduces noise compared to abstract trait summaries.
Shopping Category Prediction: The multi-hypothesis persona achieved Hit@3 = 100% and an NDCG@5 of 0.701, outperforming zero-shot LLMs and heuristic baselines in predicting held-out product categories.
Big Five Personality Prediction: The method achieved an average Mean Absolute Error (MAE) of 0.762, outperforming population norms (0.836) and closing 88% of the gap to an oracle baseline. Crucially, single-interpretation baselines produced anti-correlated predictions (negative Pearson $r$ ), while the multi-hypothesis aggregation reversed this to a positive correlation ( $r = +0.103$ ), correcting systematic "persona bias."
Shopping Attitude Alignment: The inferred personas achieved 76.6% normalized accuracy in predicting self-reported shopping attitudes (e.g., price sensitivity, research habits), outperforming single-interpretation ablations.

Significance and Claims

The paper claims that the IToM pipeline successfully extracts persona-level understanding from ordinary interaction traces without requiring qualitative data. Its significance lies in:

Scalability: It removes the bottleneck of collecting interviews and surveys, making rich user modeling feasible for production systems.
Interpretability: It produces natural-language personas that are auditable and explainable, unlike opaque latent vectors.
Modality Agnosticism: Because the output is a cognitive profile of why a user acts rather than how they interact with a specific interface, the persona transfers directly to new modalities (e.g., from 2D web to 3D XR) without retraining.
Correction of Bias: The multi-hypothesis approach is shown to be essential for accurate personality inference, preventing the system from defaulting to stereotypes when evidence is ambiguous.

The authors note limitations, including the reliance on a single domain (Amazon shopping), moderate sample size, and the fact that the XR application is a proof-of-concept requiring further controlled studies. However, the work establishes a new framework for data-efficient, generalizable, and personalized interactive systems.