Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act

Technical Summary: Train, Retrieve, or Both? A Four-Arm Head-to-Head for Correct Statutory Citation on the Ontario Residential Tenancies Act

Problem Definition
The paper addresses the specific task of guiding self-represented tenants, landlords, and help-desk staff to the correct statutory provision within the Ontario Residential Tenancies Act, 2006 (RTA) and its core regulation. The objective is not to generate fluent legal advice but to provide a precise citation (Act, section, and subsection) in response to a natural language question. The core research question posed by the operator is empirical: Is fine-tuning (SFT) sufficient for this task, or is a hybrid Retrieval-Augmented Generation (RAG) approach necessary? The study aims to minimize hallucinations (citing non-existent provisions) while maximizing exact-match citation accuracy.

Methodology
The authors conducted a four-arm head-to-head evaluation using the Qwen2.5-7B-Instruct model as the base. The study utilized a custom dataset derived from the RTA and O. Reg. 516/06, as no public question-to-citation dataset exists for this jurisdiction.

1. Data Construction:

   • Training Data: 2,148 synthetic question-to-citation pairs generated by paraphrasing statutory sections. The data was split to prevent leakage (group-aware by section/Part), resulting in a training set of 1,473, a synthetic test set of 345, and an unseen-section slice of 330.
   • Evaluation Set: A small, real-world set of 27 questions mined from tenant-law sites, explicitly citing RTA sections. This set is "human-verification-pending."
   • Corpus: The RTA and regulation were parsed into 358 provisions, with each provision serving as a single, structure-aware RAG chunk. This inventory also serves as a deterministic whitelist for hallucination checks.

2. Experimental Arms:

   • Base (Zero-Shot): The instruct model without fine-tuning or retrieval.
   • SFT-Only: A LoRA (Rank 32, $\alpha=64$) supervised fine-tuning on the synthetic pairs, without retrieval.
   • RAG-Only: The base model prompted to cite only from retrieved context using a hybrid BM25 + bge-small retriever. A "cite-from-context" guardrail acts as a hard existence check.
   • SFT+RAG Hybrid: The LoRA-fine-tuned model prompted to select the correct citation from the retrieved candidates.

3. Ablations and Variables:

   • Retrieval Depth ($k$): Tested at $k=5$ and $k=10$.
   • Retrieval Strength: Compared the cheap bge-small hybrid against a heavier pipeline using bge-large + a cross-encoder reranker.
   • Training Scale: Tested an improved/larger training set (2,645 pairs).

4. Metrics:

   • Citation Exact-Match: Requires matching instrument, section, and subsection. Section-only matches receive partial credit (0.5).
   • Hallucination Rate: Deterministic check against the pinned inventory; any citation not in the inventory is a hallucination.
   • F1: Computed over predicted vs. gold citation sets.

Key Results
The study reports results on the 27-item real evaluation set (single-run point estimates, preliminary due to small $n$).

• Base Model: Failed completely (0.00 exact-match) with an 81% hallucination rate.
• SFT-Only: Learned the citation format but struggled with specific section numbers, achieving 0.148 exact-match and reducing hallucinations to 14.8%. It failed to memorize exact section numbers effectively.
• RAG-Only: Proved essential. It achieved 0.44 exact-match (at $k=5$) and drove hallucinations to 0.00 by construction (due to the existence-check guardrail).
• SFT+RAG Hybrid: Achieved the highest score of 0.481 exact-match (0.574 F1) with 0.00 hallucinations at $k=10$.
  • The "SFT×k" Effect: Increasing retrieval depth ($k$) hurt the zero-shot RAG model (dropping from 0.44 at $k=5$ to 0.37 at $k=10$), likely due to distractors. Conversely, the SFT model improved with higher $k$ (0.333 at $k=5$ to 0.481 at $k=10$), suggesting SFT teaches robust selection from noisier, higher-recall candidate sets.
• Efficiency Findings:
  • Replacing the cheap bge-small retriever with a heavier bge-large + cross-encoder reranker did not improve performance (both RAG-only and Hybrid scored 0.37 with the heavy retriever).
  • Increasing the training data size did not yield measurable gains (0.444 vs. 0.481).

Significance and Claims
The paper provides a measured, empirical answer to the "train vs. retrieve" debate in the context of statutory citation:

1. Retrieval is Essential: Fine-tuning alone is insufficient for accurate statutory citation; retrieval is required to ground the model and eliminate hallucinations.
2. Hybrid Superiority (Conditional): The SFT+RAG hybrid performs best, primarily because SFT makes the model more robust to the "noise" of larger candidate sets that confuse zero-shot RAG models.
3. Efficiency over Complexity: For short-statute corpora like the RTA, expensive specialized retrieval models (larger embedders, rerankers) and larger training datasets do not provide measurable improvements over a lightweight bge-small hybrid and standard synthetic data.
4. Limitations and Targets: The system meets three of four success gates: it eliminates hallucinations, provides a significant lift over the base model, and answers the operator's question. However, it fails to meet the aspirational 0.70 exact-match target, stopping at 0.481. The authors attribute this gap to retrieval recall ceilings (~0.89), subsection-level difficulty, and the small size of the evaluation set.

Conclusion
The authors conclude that while the artifact successfully zeroes hallucinations and clears the lift-over-base bar, the 0.481 exact-match score on a 27-item set is preliminary. The fine-grained ordering between the retrieval-bearing arms (e.g., 0.481 vs. 0.44) is within statistical noise. The primary contribution is the demonstration that a cheap, hybrid SFT+RAG approach is sufficient for this task, outperforming both pure fine-tuning and more complex retrieval pipelines, though further work is needed to reach higher accuracy targets on a larger, human-verified evaluation set.