Technical Summary: FinPerMA

Problem Statement

Large Language Model (LLM) agents are increasingly deployed as personalized assistants in high-stakes domains like financial advising. However, it remains unclear whether these systems can maintain and update an individualized user model over long horizons. Existing personalized-memory benchmarks suffer from two critical gaps:

1. Limited Event-Conditioned State Change: Current benchmarks focus on factual retention or long-context recall but lack control over how latent user preferences evolve after consequential events. This makes it difficult to distinguish between simply recalling past facts and reasoning with an updated user state.
2. Limited Label Auditability: Many benchmarks rely on weakly constrained LLM generation for post-event trajectories, entangling the generator's properties with the evaluation target.

Furthermore, existing financial LLM resources (e.g., FinGPT, FinBen) do not center the longitudinal personalized-memory problem.

Methodology: FinPerMA Benchmark

The authors introduce FinPerMA, an event-grounded benchmark designed to evaluate personalized memory against frozen, longitudinal investor trajectories. The system is built on a controlled generation pipeline that combines deterministic, theory-informed rules with controlled LLM narration.

1. Data Generation Pipeline

The corpus consists of $N \approx 300$ personas, each with a specific event timeline, multi-session dialogues, and evaluation questions.

• Persona Synthesis: Personas are sampled from joint distributions of demographic, financial, and psychometric variables, calibrated against real retail-investor surveys (Shenzhen Stock Exchange, Asset Management Association of China). Each persona is classified into one of four Behavioral Investor Types (BITs: Passive Preserver, Friendly Follower, Independent Individualist, Active Accumulator) using rule-based mapping.
• Event Timeline Construction: The system curates 97 dated events (17 macro, 64 industry, 16 personal) from 2020–2026. Each persona receives a timeline of 5–8 events, ensuring category diversification and minimum spacing.
• The Impact Model (Core Innovation): A three-layer scaffold generates the "event reaction" ($R_{i,k}$) for each persona-event pair:
  • Layer 1 (Deterministic Rules): Constructs an ImpactConstraint object based on five sub-modules: BIT classification, event-shock coefficients (severity-based), Big-Five trait multipliers (Neuroticism/Extraversion), experience-history aggregation (age-weighted returns), and symbolic memory-reference anchoring.
  • Layer 2 (Constrained Narration): An LLM generates a candidate EventReaction (preference deltas, inner monologue, triggered biases, confidence) strictly adhering to the hard constraints from Layer 1.
  • Layer 3 (Automated Validation): Enforces hard checks on risk direction, magnitude bounds, and forbidden biases. Failed checks trigger up to three retries; if all fail, a template fallback is used.
• Dialogue Synthesis: A Task-Oriented User Simulator generates investor-advisor dialogues where preferences surface implicitly through trades and concerns. Automated quality control (deduplication, financial plausibility, multi-LLM voting) filters the output.

2. Evaluation Framework

The benchmark evaluates memory across three dimensions and four temporal checkpoints:

• Dimensions:
  • Memory Fidelity (MF): Factual recall of attributes and past behaviors.
  • Preference Reasoning (PR): Inference of implicit preferences from behavioral traces.
  • Context Adaptation (CA): Application of updated preferences in novel decision scenarios.
• Checkpoints:
  • Zero-memory: Control (no history).
  • In-timeline: Sustained retention as events unfold.
  • Post-distraction: Retention after unrelated sessions.
  • Post-Shock: A novel checkpoint testing whether the agent integrates a high-severity event into a persistently updated user model.

The final corpus contains 2,994 questions (2,494 multiple-choice, 500 open-ended) across 276 evaluated personas.

Key Contributions

1. Dynamic Personalized-Memory Benchmark: FinPerMA jointly tests stable-profile recall and event-conditioned preference adaptation, explicitly separating recall from state updating.
2. Post-Shock Checkpoint: A direct test of whether a consequential event is integrated into persistent memory, isolating the ability to update user states from simple retrieval.
3. Auditable Generation Pipeline: A hybrid system combining deterministic theory-informed rules (based on behavioral finance literature) with controlled LLM narration and automated quality screening over 97 dated events.
4. Comprehensive Empirical Evaluation: A frozen evaluation of seven frontier LLMs and up to seven memory configurations, providing detailed attribution analysis on failure modes rather than a single aggregate score.

Results

The authors evaluated seven frontier LLMs (including Qwen, Claude, Gemini, DeepSeek, GLM, and GPT variants) and various memory architectures (BM25, BGE-M3, Mem0, MemOS, Memobase) on the frozen v8gold corpus.

• Performance Gap: All models remain far from saturated. No full-context configuration exceeded ~0.47 overall accuracy or ~39% on multiple-choice questions.
• Impact of Memory: Persona history significantly lifts performance (1.5–2.4× increase) over the parametric knowledge floor, confirming that persona-specific evidence is critical. However, even with full context, models struggle with preference adaptation.
• Retrieval vs. Structured Memory: Under a fixed backbone (Qwen3.7-Max), simple retrieval systems (BGE-M3, BM25) recovered ~88% of the accuracy gap between no-memory and full-context, using only ~10% of the context tokens. Structured/profile-based memory systems (Mem0, MemOS) recovered only 54–79% of the gap.
• Failure Modes:
  • Summary-based memory often preserves factual details but loses the preference signals necessary for personalization.
  • Post-Shock Performance: The performance gap between full-context and memory systems widens significantly after a shock event, indicating a failure to update the user model.
  • Attribution: The dominant error mode is "zero-memory default," followed by single-signal over-inference and temporal misalignment. Models also show a bias toward stereotypical profiles when individual evidence conflicts with familiar types.

Significance and Claims

The paper claims that FinPerMA provides a necessary shift in benchmarking from static factual retention to dynamic, event-conditioned user modeling.

• Differentiation: Unlike prior work (e.g., PersonaMem, LoCoMo, LongMemEval) that tests continuity or fact recall, FinPerMA isolates whether an agent can integrate a consequential event into its persistent user model.
• Design Implications: The results suggest that effective memory design for agents requires a hybrid approach: separating stable attributes from mutable preferences, timestamping changes, and retrieving raw dialogue evidence rather than relying solely on summarized profiles.
• Modesty: The authors acknowledge that FinPerMA uses synthetic personas and rule-guided preference changes; therefore, benchmark accuracy does not guarantee performance with real investors. The timelines are limited to 5–8 events, and the memory-system comparison uses a single seed. Future work is needed to test longer histories and real-world longitudinal interactions.

In summary, FinPerMA demonstrates that current LLM agents are capable of retrieving user history but struggle to synthesize that history into an updated, event-responsive user model, particularly after significant shocks. Simple retrieval often outperforms complex memory architectures in preserving the nuance required for personalization.