🤖 AI

Parthenon Law: A Self-Evolving Legal-Agent Framework

This paper addresses the reliability challenges of deploying legal-domain LLM agents by presenting a large-scale empirical study on Harvey LAB and introducing \textsc{Parthenon}, a self-evolving framework that modularizes legal roles and tools to enable auditable, experience-driven improvements without modifying model weights.

Original authors: Hejia Geng, Leo Liu

Published 2026-06-04✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hejia Geng, Leo Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant, hyper-fast law student to help you with a massive legal case. This student has read every law book in the library and can write a perfect sentence in seconds. However, when you ask them to handle a whole case from start to finish, they often miss small but critical details: they forget a deadline, miscount a dollar amount, or fail to cite the specific page where a law is written.

This paper, "Parthenon Law," argues that the problem isn't that the "student" (the AI model) isn't smart enough. The problem is that the work system around them is broken.

Here is the breakdown of their solution, using simple analogies:

1. The Problem: The "Brilliant but Distracted Intern"

The authors tested the smartest AI models available on 12,510 real-world legal tasks (like reviewing contracts or analyzing court deadlines).

The Result: Even the smartest AIs could get 80–90% of the individual questions right. But in the legal world, getting 90% right isn't enough. If you miss one deadline or one citation, the whole document is useless.
The Analogy: Imagine a chef who can chop vegetables perfectly and season a steak perfectly. But if they forget to turn on the oven, the meal is ruined. The "oven" (the process) was missing, not the chef's skills.

2. The Solution: The "Parthenon" Framework

The authors built a new system called Parthenon. Instead of just asking the AI to "do the work," they built a rigid, six-layer "workshop" around the AI. Think of it like building a high-tech factory floor around a robot.

The framework has three main parts:

The "Checklist" (Skills & Tools):
Before the AI writes a single word, it is forced to use specific tools. It can't just "guess" a date; it must run a "Date Calculator" tool. It can't just "find a law"; it must use a "Search Tool" that forces it to show its work.
- Analogy: It's like giving the intern a checklist that says: "1. Check the calendar. 2. Count the money. 3. Find the source. 4. Verify the numbers." They cannot skip a step.
The "Three-Headed Monster" (Solver, Evaluator, Learner):
The system splits the work into three distinct roles that don't talk to each other in a way that causes cheating:
1. The Solver: Does the actual drafting.
2. The Evaluator: A separate "judge" that grades the draft against the rules after it's done.
3. The Learner: A mechanic that looks at the "judge's" notes and fixes the checklist or the tools for next time.
- Analogy: The Solver writes the essay. The Evaluator grades it. The Learner doesn't change the essay; instead, the Learner rewrites the instructions for the next student so they don't make the same mistake.
The "Anti-Cheating" Rule (Anti-Leakage):
This is crucial. The system learns from its mistakes, but it is strictly forbidden from memorizing the answers to the specific test questions.
- Analogy: If the intern fails a math test, the system teaches them how to do long division better. It does not teach them that "Question 5's answer is 42." This ensures the system gets smarter generally, rather than just memorizing the test.

3. The Results: "Better Process, Not Just Smarter Brains"

The authors ran the same AI models with and without this new "Parthenon" workshop.

Without Parthenon: The AI was like a fast car with no brakes. It went fast but crashed often.
With Parthenon: The AI became a reliable delivery truck. It followed the route, checked the cargo, and arrived safely.

The Magic Number: Adding this framework improved the AI's performance by about the same amount as upgrading to a much more expensive, "smarter" AI model. In fact, a cheaper AI model with the Parthenon system performed better than a top-tier AI model without it.

4. The Bottom Line: The "Co-Pilot"

The paper concludes that this system is not a replacement for human lawyers.

The Reality: Even with the Parthenon system, the AI still gets about 10% of the tiny details wrong.
The Role: The AI is now a "super-draftsman." It does 90% of the heavy lifting, checks its own work, and flags the remaining 10% for a human lawyer to review.
The Benefit: Instead of a human spending 12 hours drafting a document from scratch, they can spend 10 minutes reviewing a draft that is already 90% perfect and grounded in the actual evidence.

In short: Parthenon doesn't make the AI "smarter" in a magical way; it just forces the AI to stop guessing and start following a strict, auditable, self-improving set of rules. It turns a chaotic brainstorming session into a disciplined legal workflow.

Technical Summary: Parthenon Law: A Self-Evolving Legal-Agent Framework

1. Problem Statement

The deployment of Large Language Model (LLM) agents in legal practice faces three critical obstacles despite the potential to transform document-heavy matters into reviewable work products:

Lack of Empirical Evidence: There is no large-scale data on how current state-of-the-art model-and-harness combinations perform on end-to-end legal matters.
Architectural Mismatch: Existing agent architectures are general-purpose harnesses not adapted to the specific invariants of the legal vertical (e.g., strict deadlines, source traceability, and deliverable compliance).
Static Systems: In a domain where facts, authorities, and deadlines shift, there is no mechanism for systems to learn from their own outcomes without fine-tuning model weights or risking data leakage.

Current evaluations show that while stronger models improve per-criterion accuracy, they fail to achieve "strict matter completion" (passing all criteria for a single matter). Common failure modes include incomplete source coverage, lost quantitative details, malformed deliverables, and weak grounding. The bottleneck is identified not as model capability alone, but the absence of a structured legal work system surrounding the model.

2. Methodology: The PARTHENON Framework

The authors introduce PARTHENON, a six-layer, self-evolving legal-agent framework designed to wrap existing workspace runtimes with legal-specific controls. The architecture is organized around attribution and auditability:

2.1 Architectural Layers

Model Layer: A pluggable capability provider (e.g., GPT, Claude, Gemini) that allows routing matters by practice area without binding the system to a single model.
Harness Layer: The observable execution contract (e.g., Codex, Claude Code, OpenCode) providing workspace, tool access, and trace capture. PARTHENON treats this as pluggable, wrapping it with legal specialization.
Agent Layer: Defines strict role boundaries to prevent information leakage:
- Solver: Drafts the work product using task, sources, skills, and tools.
- Evaluator: Scores the finished work against a rubric outside the solver's context to prevent memorization.
- Learner: Proposes task-agnostic edits to the harness based on redacted traces and aggregate signals.
Knowledge Layer: Stores durable legal memory (statutes, deadlines, schemas, calendars, synonyms) as data, not prompt text. These are general-purpose objects, explicitly excluding matter-specific facts or benchmark answers to prevent leakage.
Tools Layer: Converts recurring legal requirements into deterministic, interpretable operations (e.g., date arithmetic, citation checks, number reconciliation). These replace implicit model memory with executable code.
Skills Layer: Contains rubric-blind procedural plans (e.g., triage, issue lifecycle, mandatory tool invocation) selected by matter class. This layer turns empirical failures into reusable procedures.

2.2 The Self-Evolving Loop

PARTHENON implements a gated optimization loop that updates the harness rather than model weights:

Execution: The Solver produces a draft.
Evaluation: The Evaluator scores the draft against a hidden rubric, generating feedback.
Learning: The Learner receives redacted failure trajectories (stripped of task IDs, rubric phrases, and client data) and proposes edits to the Knowledge, Tools, or Skills layers.
Gating: Edits are admitted only if they are generalizable, pass static safety checks, and strictly improve the per-task pass rate. This "anti-leakage" protocol ensures the system learns procedural improvements rather than memorizing benchmark signals.

3. Experimental Setup

Benchmark: Harvey LAB, a corpus of 1,251 matters across 24 practice areas, featuring source documents, deliverables, and expert rubrics.
Baselines: Evaluated across four execution families: Direct API prompting, a basic legal-native harness, and the Codex and Claude Code workspace harnesses.
Models: Tested across three model tiers (GPT-5.4-mini, GPT-5.5, and Claude Sonnet 4.6/Haiku 4.5).
Metrics:
- Criterion Accuracy: The share of all rubric criteria passed.
- All-Pass: The strict share of matters where every criterion is passed.

4. Key Results

4.1 Performance Lift

With the model and agent harness fixed, adding PARTHENON yields performance gains comparable to upgrading the base model:

Accuracy Gains: PARTHENON increased pooled criterion accuracy by +13.8, +10.2, and +7.4 percentage points across the three model tiers, reaching 82.0%, 89.9%, and 90.2% respectively.
Strict Completion: On weaker solvers, strict all-pass completion roughly tripled (e.g., from 14 to 42 matters for GPT-5.4-mini).
Error Reduction: The framework significantly reduced mechanical errors (missing facts, numbers/dates, deliverable form) which previously dominated baseline failures.

4.2 Mechanism of Improvement

Action Mix: The improvement is driven by a new "tool/script" bucket of actions (e.g., mandatory audits, schema fills) rather than increased text generation. Baseline agents were read-heavy; PARTHENON agents perform structured inspections and validations.
Cost Efficiency: For the most expensive solver (GPT-5.5), PARTHENON actually reduced cost per matter ( $1.51 →$ 1.29) while increasing accuracy, as the audit loop produced shorter, more precise outputs. For cheaper solvers, the cost increase was minimal relative to the accuracy gain.
Ablation Studies:
- Optimization: The self-improvement loop converges to a transferable harness, with different solvers converging to similar accuracy levels.
- Reasoning Effort: Increasing raw inference budget (reasoning effort) yielded unreliable results, whereas procedural controls provided dependable gains.
- Document Summaries: Adding cached document summaries did not improve performance on long inputs; the bottleneck was procedural discipline, not input length.

4.3 Human Comparison

Against a "release standard" approximating human lawyer review (100% accuracy):

Accuracy: Even the strongest PARTHENON configuration (90.2% criterion accuracy) passes every criterion on only ~12% of matters, indicating it is a draft assistant, not an autonomous lawyer.
Time & Cost: The system offers a massive efficiency gain. Estimated time per matter drops from ~12.6 hours (human) to ~10 minutes (AI), and cost drops from ~$4,399 to ~$0.81.

5. Significance and Claims

The paper claims that the primary barrier to reliable legal AI is procedural, not parametric. Stronger models fail legal work because they lack a structured system to enforce professional invariants (deadlines, citations, grounding).

Key Contributions:

Empirical Analysis: A large-scale study of 12,510 agent trajectories on Harvey LAB, revealing that strict matter completion remains low even for frontier models.
PARTHENON Framework: A six-layer architecture that separates model capability from legal memory, tools, and procedural skills, making failures auditable and editable.
Self-Evolving Loop: A mechanism to convert scored failures into task-agnostic harness updates without fine-tuning models or leaking benchmark data.

Conclusion:
The authors conclude that PARTHENON transforms the role of legal AI from "drafting from scratch" to "reviewing a source-grounded, audit-flagged first draft." By wrapping solvers in an auditable legal harness, the system achieves gains comparable to model upgrades and transfers across different model families. The significance lies in demonstrating that reliability in high-stakes domains can be achieved through external, inspectable procedural controls rather than relying solely on increasing model scale.