When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval

Technical Summary: When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval

Problem Statement
Legal case retrieval remains a challenging task due to the complexity of legal language, the length of documents, and the necessity for precise lexical alignment between user queries and relevant cases. While dense retrieval models based on neural embeddings have advanced significantly, empirical studies on the Chinese legal benchmark LeCaRD-v2 indicate that they often underperform compared to traditional lexical matching methods. Specifically, BM25 consistently outperforms representative dense retrieval models (including those fine-tuned on legal data) across various recall cutoffs. This suggests that rather than replacing BM25, a more promising direction is to enhance its performance by improving the lexical alignment of queries through rewriting. However, designing high-quality, rule-driven query rewriting strategies typically requires substantial domain expertise, and naive rule generation often leads to suboptimal results.

Methodology
The authors propose a self-evolution framework for rule-driven query rewriting that operates without any gradient-based parameter training. The system employs an LLM-based agent that interacts with an automated evaluation environment to iteratively refine a set of rewriting rules. The framework functions as a closed-loop system where adaptation emerges from interaction rather than model optimization.

The agent's decision-making process involves three primary actions:

1. Rule Generation: The agent creates new query-rewriting rules by analyzing the current active rule set, previously eliminated rules, and accumulated experimental scores. It can either modify existing rules to amplify effective behaviors or propose entirely new strategies, ensuring new rules do not degenerate into previously discarded patterns.
2. Experiment Planning: The agent selects subsets of active rules to form combinations for testing. It leverages historical experimental results to identify promising combinations (e.g., those with complementary effects) while avoiding redundant evaluations of previously tested sets.
3. Rule Elimination: To prevent the strategy space from becoming unmanageable, the agent identifies and removes ineffective rules. This process utilizes a two-stage consensus mechanism to mitigate stochastic errors: the agent first determines if a rule should be removed based on multiple independent trials, and if a consensus is reached, it selects a specific rule for elimination based on a voting threshold.

The environment executes these decisions by invoking a frozen external LLM to rewrite queries according to the selected rules, which are then evaluated using the BM25 algorithm on a development set. The agent iterates through this process (up to 500 steps) to refine its rule set before being evaluated on a held-out test set.

Key Contributions

• Re-evaluation of Baselines: The work demonstrates that BM25 remains a strong baseline for legal case retrieval, outperforming several state-of-the-art dense retrieval models on LeCaRD-v2, and advocates for enhancing BM25 via query rewriting rather than replacement.
• Self-Evolution Framework: The authors introduce a novel, training-free framework where an agent autonomously optimizes rewriting rules through generation, experiment planning, and elimination. This approach offers interpretability and controllability, which are critical in the legal domain.
• Behavioral Analysis: The paper provides a detailed investigation into the mechanisms driving self-evolution, characterizing the behavioral dynamics of LLM agents and identifying intrinsic factors that influence their capacity for self-improvement.

Experimental Results
The framework was evaluated on the LeCaRD-v2 benchmark. Key findings include:

• Performance vs. Baselines: When powered by a high-capacity core LLM (specifically gpt-oss-120b), the self-evolution framework consistently outperformed non-evolutionary baselines, including human-designed rules and a greedy rule selection strategy. It achieved an average recall improvement of +0.61% over the greedy strategy and +1.03% over manually designed rules.
• Model Dependency: The effectiveness of the framework is highly dependent on the core LLM's capabilities. Smaller models (e.g., Qwen3-4B-Thinking) failed to consistently outperform greedy strategies, often over-optimizing for higher recall cutoffs (e.g., Recall@500) while underperforming at lower cutoffs.
• Mechanisms of Success: Detailed analysis revealed that the success of self-evolution relies less on the progressive generation of increasingly effective individual rules (which fluctuated in quality) and more on the agent's ability to:
  • Plan Experiments: Effectively leverage historical results to select high-quality "anchor" combinations for new experiments.
  • Eliminate Rules: Accurately identify and discard ineffective rules while retaining effective ones, a task where larger models demonstrated better control over the active rule set size.

Significance and Claims
The paper claims that self-evolving agents offer a viable path for the interpretable and adaptive enhancement of legal case retrieval systems without the need for extensive training data or model fine-tuning. The authors highlight that the framework's success is contingent upon the core LLM's reasoning capabilities, specifically its ability to plan experiments based on historical data and to make conservative, evidence-driven decisions regarding rule elimination.

The work acknowledges limitations, noting that the approach relies heavily on the core LLM's ability to follow complex instructions and generate coherent text in the target language (Chinese). Models lacking these specific capabilities failed to generate meaningful rules or adhered poorly to format constraints. Furthermore, the authors suggest that for application in different legal jurisdictions, appropriate seed rules should be provided during initialization to account for variations in legal terminology and document structure. The study does not claim to replace dense retrieval entirely but positions self-evolving rule-based rewriting as a robust, complementary strategy to enhance existing lexical retrieval systems.