Technical Summary: Credit Cards, Confusion, Computation, and Consequences

Problem Statement

Despite the critical role of financial literacy in preventing excessive debt and bankruptcy, U.S. financial literacy rates have stagnated near 50% for nearly a decade. While Large Language Models (LLMs) are increasingly adopted by consumers for personal finance guidance, existing benchmarks for financial reasoning primarily focus on stock market analysis, investment strategies, or professional corporate finance tasks. There is a significant gap in evaluating whether LLMs can perform the numerical reasoning required for real-world financial literacy, specifically the ability to interpret complex, conditional, and exception-heavy legal documents like credit card agreements. These documents govern everyday consumer decisions regarding fees, interest accrual, and penalties, yet they are notoriously difficult for humans to interpret and untested for AI reliability.

Methodology

Dataset Construction: CREDITCARDQA

The authors introduce CREDITCARDQA, the first benchmark for financial literacy numerical reasoning derived directly from real-world credit card agreements.

• Source Material: 27 credit card agreements from major issuers (e.g., American Express, Discover, Barclays) covering premium, everyday, and subprime products, representing over 80% of total outstanding balances.
• Processing: Agreements were converted from PDF to Markdown using the Marker tool and manually verified.
• Question Generation: A team of finance and ML experts generated 1,800 question-answer pairs. Questions were derived from 280 extracted financial numeric terms (e.g., APRs, fees, grace periods).
• Structure: The dataset includes 1,308 third-person questions and 492 first-person variants to reflect natural consumer inquiries. Questions require an average of 4.4 mathematical operations and often involve conditional logic (e.g., "greater than," "if/then"), comparisons, and monetary constraints.
• Split: The dataset is divided into a development set (800 questions with full annotations) and a held-out test set (1,000 questions).

Evaluation Framework

The study evaluated 11 models, including both Large Language Models (LLMs) and Large Reasoning Models (LRMs), spanning open-weight (e.g., DeepSeek-R1, Qwen-QwQ, Llama-3.3) and closed-source (e.g., GPT-5, Gemini 3.0 Pro) systems.

• Prompting Strategies: Models were tested under Chain-of-Thought (CoT) and Program-of-Thought (PoT) prompting. PoT requires models to generate executable Python code to perform calculations, separating reasoning from arithmetic.
• Metrics: Accuracy was measured at two numerical tolerances: ±0.2% (strict) and ±5% (lenient). Categorical answers were manually verified to handle unit mismatches and semantic equivalence.

Analysis Techniques

• Error Analysis: A fine-grained audit of 50 incorrect responses from the strongest open-weight model (GPT-OSS-120B) to categorize failure modes.
• Regression Modeling: A logistic regression was employed to isolate the impact of specific features (structural reasoning, linguistic framing, input types) on model correctness, controlling for agreement-specific difficulty.

Key Contributions

1. CREDITCARDQA Benchmark: A novel dataset of 1,800 questions grounded in real legal contracts, moving beyond standardized financial formulas to capture the complexity of consumer credit terms.
2. Empirical Evaluation of Financial Literacy: The first systematic assessment of LLMs and LRMs on financial literacy tasks, revealing that while models can perform arithmetic, they struggle with the interpretation of contractual rules.
3. Prompting Strategy Insights: Demonstration that Program-of-Thought (PoT) prompting consistently outperforms Chain-of-Thought (CoT), particularly for models with weaker baseline reasoning, effectively narrowing the performance gap between open- and closed-source systems.
4. Error Characterization: Identification that failures are rarely due to arithmetic errors but rather stem from misapplied financial rules, missed conditional logic (e.g., ignoring "if" clauses for penalties), and misunderstandings of problem intent.
5. Difficulty Factors: Quantification of what makes financial questions difficult, identifying that comparison operations, conditional branching, and monetary constraints are the primary drivers of model failure.

Results

• Overall Performance: Even the top-performing models (GPT-OSS 120B, Gemini 3.0 Pro, GPT-5) achieved accuracy rates around 70–80% under the ±5% tolerance with PoT prompting. This indicates that current models are not yet fully reliable for high-stakes financial advice.
• PoT vs. CoT: PoT yielded consistent performance gains across all models, with improvements ranging from 0.9% to 6.2% at the ±5% tolerance. The gains were most pronounced for models with weaker CoT baselines (e.g., Llama-3.3, DeepSeek-R1).
• Error Distribution: In the error analysis of GPT-OSS-120B:
  • Formula Application Errors (70%): The most common failure, where models identified the correct concept but applied the wrong rule (e.g., using simple interest instead of compounding).
  • Missing Conditions (54%): Failure to apply critical "if/only when" constraints, such as minimum fees or eligibility thresholds.
  • Numerical Calculation Errors (14%): Surprisingly low, suggesting that once rules are identified, models can compute accurately.
• Regression Findings:
  • Comparison Operations: The presence of comparison logic (e.g., "greater than") reduced the odds of a correct answer by approximately 74%.
  • Linguistic Framing: First-person phrasing ("How much will I owe?") significantly improved performance compared to third-person phrasing, suggesting a "conversational advantage" aligned with RLHF training distributions.
  • Monetary Values: Questions involving explicit monetary quantities were significantly harder to answer correctly, likely due to the complexity of interacting with caps and penalty triggers.

Significance and Claims

The paper posits that the "democratization" of financial well-being via AI is currently hindered by a brittleness in reasoning, particularly regarding the nested conditional logic found in legal contracts. The authors claim that:

1. High-Stakes Implications: Errors in financial literacy reasoning disproportionately affect financially vulnerable individuals, as mistakes often occur in edge cases like late-payment penalties or small-balance scenarios.
2. Limitations of Current Models: Current LLMs exhibit a form of "overgeneralization," applying default rules even when specific contractual exceptions (e.g., penalty APRs) should apply.
3. Path Forward: The authors suggest that Program-of-Thought (PoT) frameworks are essential for narrowing logic-processing bottlenecks. Furthermore, they advocate for "LLM-accessible" disclosures, arguing that making credit card terms more interpretable by language models is necessary to ensure automated tools provide accurate guidance.
4. Research Direction: The findings highlight that financial reasoning is not merely a mathematical challenge but a domain-specific interpretation task requiring precise adherence to conditional rules, a capability that current benchmarks and models have yet to fully master.