Technical Summary: RT-SIM – A Simulation Framework for Evaluating Large Language Models for Trade Reconstruction and Reconciliation

1. Problem Statement

Financial institutions face persistent operational risks stemming from misbooked trades, communication failures, and discrepancies between front-office executions and back-office records. While global equity markets trade over USD 500 billion daily, traditional reconciliation processes rely heavily on structured transaction records, often leaving a window of vulnerability where errors or unauthorized positions remain undetected until settlement.

A critical limitation of current approaches is the exclusion of unstructured trader communications (chats, electronic messages, recorded conversations). Historical control failures, such as the Barings Bank collapse, demonstrate that discrepancies often persist because relevant information is fragmented between booking systems and trader dialogue. Although Large Language Models (LLMs) have shown promise in extracting structured data from financial documents, there is limited evidence regarding their ability to reconstruct executed trades from imperfect, noisy, or ambiguous trader communications. The core challenge is determining whether LLMs can reliably infer trade details from conversational data when that data is degraded by noise, ambiguity, or contradictory information.

2. Methodology: The RT-SIM Framework

To address this gap, the authors introduce RT-SIM, a simulation framework designed to evaluate LLMs as trade reconciliation agents under controlled communication conditions. The framework operates as a continuous reconciliation process involving the following steps:

1. Trade Generation: A "ground truth" trade is generated from market data and stored in a Truth Book.
2. Booking: The trade is recorded in a Booked Book, with optional error injection to simulate mis-booking, rogue trading, or operational failures.
3. Communication: Trader dialogue is generated via rule-based templates or LLMs. Crucially, the framework supports Anchored (dialogue conditioned on the executed trade) and Unanchored (dialogue generated independently) modes to control the alignment between communication and reality.
4. Inference: A Trading Assistant (TA), powered by an LLM, processes the dialogue transcript to extract structured trade information (buyer, seller, quantity, price) and records it in a TA Book.
5. Reconciliation: The system compares the Truth, Booked, and TA records to identify discrepancies (missing trades, unsupported trades, reversed entries) and recalculates Profit and Loss (P&L) impacts.

Experimental Design

The study evaluated four communication conditions:

• Competent: Clear, explicit trade communication.
• Noisy: Inclusion of irrelevant conversational content and chatter.
• Confused: Introduction of ambiguity and contradictory information.
• Semantically Degraded (Gibberish): Dialogue containing no meaningful trade information (control condition).

Experiments utilized 14 model pairings across OpenAI (GPT-4o-mini, GPT-3.5 Turbo), Anthropic (Claude Haiku 4.5), and Google (Gemini 2.5 Flash Lite) providers. Performance was measured using Match Rate, Precision, False Negative Rate (FNR), and False Positive Rate (FPR), alongside operational metrics like latency and cost.

3. Key Contributions

The paper makes three primary contributions to the literature on digital finance and AI:

1. RT-SIM Framework: A reproducible simulation environment that generates synthetic trading activity and trader dialogue to systematically assess how communication quality influences reconciliation performance.
2. LLM-Based Reconciliation Architecture: A novel framework that utilizes unstructured trader communications as an independent source of evidence for reconciliation, enabling continuous monitoring of discrepancies between conversational records and official books.
3. Empirical Evidence on Communication Degradation: A comprehensive evaluation of LLM performance across competent, noisy, confused, and semantically degraded environments, providing data on the robustness and limitations of current frontier models in financial operations.

4. Results

The experiments yielded distinct findings regarding model capability and environmental factors:

• Baseline Performance: Under competent communication, frontier models (GPT-4o-mini and Claude Haiku 4.5) achieved high reconstruction accuracy, with match rates exceeding 95%. Cross-provider evaluations showed strong robustness; for instance, a Claude Haiku 4.5 TA reconstructed 98.8% of trades generated by GPT-4o-mini. In contrast, Gemini 2.5 Flash Lite and GPT-3.5 Turbo exhibited significantly lower accuracy.
• Impact of Noise: Irrelevant conversational noise had a negligible effect on performance. Models maintained match rates above 97% in noisy conditions, indicating they can effectively filter out non-trade-related chatter when trade information remains explicit.
• Impact of Ambiguity: Performance deteriorated significantly under confused (ambiguous/contradictory) conditions. Match rates dropped by over 50 percentage points for many configurations. Notably, models tended to become conservative, resulting in high False Negative Rates (missing trades) rather than generating false positives (hallucinating trades).
• Semantic Degradation: In the gibberish condition, reconstruction performance collapsed to 0% across all models. No valid trades were reconstructed, confirming that success depends on interpretable semantic signals rather than superficial linguistic patterns.
• Operational Feasibility: The cost of reconstruction was extremely low (approximately $0.0002 per trade for GPT-4o-mini), and inference latency was under one second, suggesting economic and technical viability for large-scale deployment.

5. Significance and Claims

The paper claims that RT-SIM demonstrates the feasibility of using LLMs to transform unstructured trader communications into a structured, additional source of reconciliation evidence. The significance of these findings lies in several areas:

• Enhanced Control Mechanisms: By inferring trades from dialogue, institutions can create an independent ledger to validate recorded transactions, potentially identifying discrepancies (such as unbooked positions or unsupported transactions) much earlier than traditional batch processes allow.
• Risk and Fraud Detection: The framework supports the identification of control failures, including off-book trading and fictitious transactions, by highlighting mismatches between what was communicated and what was booked.
• Communication Quality as a Determinant: The study establishes that communication quality is a more critical determinant of success than model capability alone. While models are robust to noise, they are highly sensitive to ambiguity. Therefore, the effectiveness of LLM-based reconciliation is contingent upon the clarity of the underlying communication environment.
• Practical Deployment: The authors conclude that while computational costs are negligible, successful deployment requires integration with existing governance frameworks and operational controls, rather than relying solely on model inference. The technology is presented not as a replacement for existing systems, but as a complementary layer for continuous monitoring and discrepancy detection.

The paper maintains a modest tone, acknowledging that the results are derived from a synthetic environment and that real-world deployment would require further validation with proprietary data and integration into complex organizational workflows.