From Forecasts to Auditable Reports: Evidence Contracts for LLM-Assisted Housing-Guarantee Risk Monitoring

Technical Summary: From Forecasts to Auditable Reports

Problem Setting
Public housing guarantee institutions face a critical challenge in converting noisy, next-month risk forecasts into timely, auditable operational reports. In the context of South Korea's jeonse (large upfront deposit) guarantee market, this task is complicated by three factors: (1) the sparsity of upper-tail risk events (guarantee accidents), which makes average-error minimization insufficient for risk monitoring; (2) data governance constraints that prevent the transmission of raw, confidential guarantee records to external Large Language Models (LLMs); and (3) the risk of "hallucination," where fluent LLM narratives may invent evidence, distort numerical units, or fabricate historical analogues. The core problem is not merely prediction, but evidence-constrained risk communication: generating reports that are verifiable, grounded in specific data, and suitable for analyst review without exposing sensitive source data.

Methodology
The authors propose an evidence-constrained reporting pipeline that decouples statistical forecasting from narrative generation, inserting structured "evidence contracts" between the two.

1. Risk-Sensitive Forecasting Backbone:

   • The system employs a Temporal Fusion Transformer (TFT) to predict next-month segment-level guarantee accident rates.
   • Unlike standard models optimized for Mean Absolute Error (MAE), the TFT is trained with a regret-style objective. This asymmetric loss function imposes a higher penalty for under-predicting high-risk (right-tail) events than for over-predicting low-risk segments, aligning the model with the operational costs of missed warnings.
   • Crucially, the TFT exposes internal variable-selection and temporal-attention weights, which serve as the "rationale" for the prediction.

2. Model-Reasoning Retrieval:

   • To find relevant historical precedents without leaking raw data, the system constructs a model-reasoning representation ($Z_{i,t}$) by concatenating the TFT's variable-selection and temporal-attention matrices.
   • Centered Kernel Alignment (CKA) is used to measure similarity between the current window's reasoning matrix and historical windows. This retrieves "analogous cases" based on how the model reasoned (variable-time patterns) rather than just raw covariate values.
   • High-risk reference windows are selected separately based on realized outcomes to ensure the system distinguishes between "similar reasoning" and "severe historical outcomes."

3. Evidence Contracts and Structured Reporting:

   • Before an LLM is invoked, a deterministic module compiles a Typed Evidence Contract ($E_{i,t}$). This contract contains fixed fields: the forecast, baseline, direction label (Up/Down/Flat), key drivers, temporal evidence, analogous cases, high-risk references, and caveats.
   • The LLM is strictly constrained to verbalize the contents of this contract. It is prohibited from generating new forecasts, drivers, or historical analogues.
   • Three interfaces are tested: Raw Data (unstructured context), Evidence Card (typed contract added), and Structured Report (contract + fixed report grammar).

4. Audit Protocol:

   • A deterministic audit layer verifies the generated draft against the evidence contract. It checks for direction consistency, numerical fidelity, unit correctness, feature validity, and the absence of unsupported inferences.
   • The output is a set of flags indicating which claims require analyst verification, rather than an automated pass/fail decision.

Key Contributions

1. Formulation: The paper reframes housing-guarantee monitoring as a process of converting sparse risk forecasts into auditable review evidence under capacity and governance constraints, rather than a pure prediction task.
2. Workflow: It introduces an evidence-construction workflow that links forecast values, model-derived drivers, influential periods, and historical precedents before report generation, ensuring the LLM operates on a bounded set of admissible facts.
3. Evaluation: The study evaluates the pipeline through a multi-faceted lens: tail-sensitive forecasting utility, report fidelity (grounding), auditability, and a practitioner assessment involving 51 analysts.

Results

• Forecasting Performance: The regret-trained TFT significantly outperforms standard models (LSTM, DLinear, LightGBM) in detecting high-risk events. While the LSTM achieved the lowest MAE (0.027), it had a P90 recall of only 0.080. The proposed TFT-Regret model achieved a P90 recall of 0.560 (surfacing 14 of 25 realized high-risk cases), at the cost of a modest increase in MAE (0.041).
• Report Quality: Across eight LLMs, the use of structured evidence contracts consistently improved report quality, numerical fidelity, and claim-level grounding compared to raw data inputs.
• Practitioner Validation: In a study with 51 analysts and domain professionals, the majority rated the evidence-constrained reports as practically useful for real-world review and decision-making. Most participants endorsed an operational pilot, noting that the reports supported the necessary "analyst-in-the-loop" workflow.

Significance and Claims
The paper argues that reliable LLM-assisted reporting in high-stakes financial monitoring cannot rely on predictive models alone. Instead, it requires a coupling of predictive models with structured evidence, explicit verification, and analyst oversight.

The authors claim that their approach successfully transforms open-ended generation into bounded, verifiable reporting. By treating model outputs as "audit-oriented evidence" rather than causal explanations, and by enforcing a strict evidence contract before generation, the system mitigates the risks of hallucination and data leakage. The findings suggest that for operational risk monitoring, the value of an AI system lies not in its ability to generate fluent text, but in its ability to organize and present admissible, verified evidence that supports human judgment. The system is designed to assist, not replace, the analyst, ensuring that the final risk judgment remains a human responsibility supported by a transparent, evidence-based trail.