Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation

1. Problem Statement

Current recommendation benchmarks primarily evaluate Large Language Models (LLMs) based on behavioral imitation: whether the model's output matches historical user choices (e.g., clicks, ratings, or trades). In the financial domain, this approach is fundamentally flawed because:

• Noisy Actions: Investor decisions are often driven by short-term market volatility, emotions, or transient constraints rather than stable long-term goals.
• Misalignment of Truth: Treating observed user behavior as the "ground truth" conflates mimicking noise with providing high-quality financial advice. A model that perfectly mimics a user's impulsive trades may fail to serve their long-term utility.
• Lack of Utility Grounding: Existing benchmarks lack a mechanism to evaluate recommendations against a user's latent risk preferences and rational utility, making it impossible to distinguish between rational analysis, behavioral mimicry, and market momentum chasing.

2. Methodology: The Conv-FinRe Framework

The authors introduce Conv-FinRe, a conversational and longitudinal benchmark designed to evaluate LLMs beyond simple behavior matching.

A. Task Formulation

The task simulates a longitudinal interaction between a personalized investment advisor (the LLM) and a user over a fixed horizon ($T$). Unlike standard benchmarks with a single "gold" label, Conv-FinRe utilizes a Multi-View Alignment framework with four reference views:

1. User Choice ($y_{user}$): The empirical stock selected by the human participant (noisy behavior).
2. Rational Utility ($y_{util}$): The theoretically optimal ranking based on a calibrated utility function balancing return and risk.
3. Market Momentum ($y_{mom}$): A profit-oriented ranking based purely on recent cumulative returns.
4. Risk Sensitivity ($y_{safe}$): A conservative ranking penalizing volatility and downside risk according to the user's specific profile.

The LLM must synthesize these conflicting signals to generate a final ranking, effectively diagnosing whether it relies on rational analysis, mimics noise, or chases trends.

B. Latent Preference Grounding via Inverse Optimization

Since the model does not have access to the user's true utility function, the benchmark constructs a latent preference signal using Inverse Optimization:

• Utility Function: Defined as $U(s) = \tilde{\mu} - \lambda \tilde{\sigma}^2 - \gamma \tilde{Drawdown}$, where $\lambda$ and $\gamma$ represent the user's sensitivity to volatility and downside risk.
• Parameter Estimation: The model estimates user-specific parameters $(\lambda_i, \gamma_i)$ by minimizing the Regularized Negative Log-Likelihood of the user's observed longitudinal decision trajectory ($H_{1:T}$).
• Reference Construction: These estimated parameters are used to generate the $y_{util}$ and $y_{safe}$ reference rankings, which serve as the ground truth for "rational" decision quality without exposing the function to the LLM during inference.

C. Data Collection and Simulation

• Stock Universe: A curated set of 10 S&P 500 stocks covering 11 GICS sectors, stratified by market beta (Low, Moderate, High volatility).
• User Profiles: 10 participants provided static profiles via a structured questionnaire (aligned with MiFID II and FINRA guidelines) and interacted with a custom asset simulation tool over a 30-day horizon.
• Conversation Simulation: The benchmark generates structured, multi-turn dialogues consisting of:
  • Onboarding: A 4-turn interview to elicit financial background and risk tolerance.
  • Longitudinal Advisory: 23 steps of interaction where a panel of three specialized advisors (Rational, Momentum, Risk) offers conflicting advice, and the user makes a choice.
• Scale: 230 total instances, averaging 15 turns per instance, with substantial context complexity (avg. 4,320 tokens).

D. Evaluation Metrics

The framework employs a multi-dimensional evaluation suite:

• Utility-based NDCG (uNDCG): Measures alignment with the latent rational utility ($y_{util}$).
• MRR & Hit Rate (HR@K): Measures alignment with the observed user choice ($y_{user}$).
• Expert Alignment Score (EAS): Uses Kendall's $\tau$ to measure alignment with the three expert strategies (Utility, Momentum, Risk).

3. Key Contributions

1. Conv-FinRe Benchmark: The first conversational and longitudinal benchmark for stock recommendation that evaluates LLMs based on investor-specific utility rather than just behavioral matching.
2. Multi-View Diagnostic Framework: A novel approach using inverse optimization to disentangle behavioral alignment (mimicking user noise) from rational decision quality (optimizing for long-term utility).
3. Systematic Evaluation: A comprehensive assessment of state-of-the-art LLMs (including GPT-5.2, GPT-4o, DeepSeek, Qwen, and Llama variants) revealing distinct advisory behavior patterns.

4. Experimental Results

The evaluation of 8 diverse LLMs revealed a persistent tension between rational decision quality and behavioral alignment:

• Rational vs. Behavioral Trade-off:
  • Models like Llama-3.3-70B achieved the highest uNDCG (0.97), indicating strong alignment with rational utility, but had lower Hit Rates (mimicking user choices less effectively).
  • Models like Qwen2.5-72B and Llama3-XuanYuan3-70B excelled in MRR and Hit Rate, effectively mimicking the user's noisy, realized decisions, but showed lower alignment with the idealized rational utility.
• Expert Alignment Patterns:
  • Llama-3.3-70B showed strong coupling between Rational Utility and Market Momentum alignment but struggled significantly with Risk Sensitivity (Safety), suggesting it conflates growth signals with safety.
  • DeepSeek-V3.2 demonstrated the most balanced profile, maintaining stable alignment with Safety while avoiding extreme bias toward return-driven metrics.
  • Domain-Specific Models (XuanYuan3): Despite lower expert alignment scores, they achieved high behavioral hit rates, acting more like "seasoned human consultants" who prioritize pragmatic, albeit noisy, investor preferences over rigid mathematical formulas.
• Preference Discovery Dynamics:
  • Adaptive Advisors (e.g., GPT-5.2, DeepSeek): Showed significant utility alignment gains when conversational history was available, successfully extracting latent risk preferences.
  • Behavioral Overfitters (e.g., Qwen2.5, XuanYuan3): Experienced degraded utility alignment when history was introduced, indicating they overfit to transient user actions rather than inferring stable preferences.

5. Significance and Impact

• Paradigm Shift: Conv-FinRe challenges the prevailing "behavior-as-truth" paradigm in recommendation systems, arguing that in finance, mimicking user behavior can be detrimental if that behavior is irrational or short-sighted.
• Diagnostic Capability: The benchmark provides a tool to diagnose why a model fails—whether it is overfitting noise, blindly chasing market momentum, or failing to understand risk tolerance.
• Future Directions: The findings suggest that future financial AI should not aim solely for behavioral alignment but must navigate the trade-off between providing the most rational advice and maintaining empathetic alignment with user intent. The dataset and codebase are publicly released to facilitate further research in utility-grounded financial AI.