Technical Summary: MPC-Patch-Bench

1. Problem Statement

Current repository-level benchmarks for evaluating Large Language Model (LLM) code repair, such as SWE-bench, are structurally inadequate for Secure Multi-Party Computation (MPC) software. The authors identify three primary structural failures when attempting to transplant general-purpose benchmarks to the MPC domain:

1. Dominance of Generic Infrastructure: MPC repositories are largely composed of generic Python infrastructure (CI/CD, API wrappers) rather than core cryptographic logic, causing naive benchmarks to measure standard software engineering rather than MPC-specific reasoning.
2. Lack of Standardized Tests: High-value MPC fixes often lack the standardized "Fail-to-Pass" (F2P) and "Pass-to-Pass" (P2P) tests required by rigid extraction pipelines. For instance, applying strict SWE-bench filtering to 7,305 raw MPC pull requests (PRs) yielded only 42 viable instances.
3. Insufficiency of Functional Evaluation: Standard functional correctness (passing tests) is insufficient for MPC. An MPC patch must also satisfy cryptographic safety guarantees (e.g., data-obliviousness, secret sharing integrity) and numerical fidelity (avoiding finite-ring or fixed-point drift), which standard unit tests cannot detect.

Consequently, existing MPC-specific efforts are limited to operator-level synthesis or single-framework tasks, leaving a gap in evaluating LLM agents on real-world, repository-level MPC repair.

2. Methodology

The paper introduces MPC-Patch-Bench, a repository-level benchmark comprising 205 fully verified instances curated from five major MPC frameworks: CrypTen, tf-encrypted, MP-SPDZ, SecretFlow, and PySyft. The construction and evaluation rely on two core frameworks:

A. Data Curation Framework

This framework converts raw PRs into executable benchmark instances through a two-stage process:

1. Domain-Specific Curation Agent: An AI agent filters 7,305 raw PRs through three cryptographic layers to retain only those exercising genuine MPC reasoning:
   • Layer 1 (Primitive Check): Verifies manipulation of core primitives (e.g., secret sharing, Oblivious Transfer).
   • Layer 2 (Protocol Logic): Distinguishes fixes addressing mathematical MPC correctness from generic system wrappers.
   • Layer 3 (Domain Specificity): Ensures the resolution requires specialized constraints (e.g., data-obliviousness, fixed-point truncation management).
   • Routing: Candidates are routed based on confidence scores; high-confidence PRs are accepted automatically, while medium-confidence ones undergo human review. This reduced the pool to 1,175 cryptographically relevant candidates.
2. Human-AI Completion Engine: To address the lack of standardized tests in MPC PRs, a human expert co-pilots with a synthesis agent. The human provides domain intuition and verifies cryptographic correctness, while the agent synthesizes missing problem statements, F2P/P2P test patches, and execution harnesses. This process rescued high-value but incomplete PRs, expanding the final dataset from the 42 instances yielded by strict filtering to 205 fully verified instances.

B. The MPC Verifier

To evaluate LLM-generated patches, the authors designed a dual-stream verifier that goes beyond functional testing:

1. Dynamic Testing Stream: Executes the secure protocol against a plaintext oracle to detect numerical-fidelity drift (precision errors) and validates algebraic edge cases via differential testing.
2. Domain-Specific Static Application Security Testing (SAST): Performs source-code scanning to flag MPC anti-patterns, including:
   • Unsafe reveals (e.g., reveal() inside loops).
   • Insecure arithmetic (e.g., raw floating-point coercion on secret-shared types).
   • Illegal type casting (unauthorized public/private conversions).
   • Context-aware crypto-rule violations.

3. Key Contributions

• MPC-Patch-Bench: The first repository-level benchmark specifically designed for evaluating LLM code repair in MPC software, containing 205 diverse, executable instances across five frameworks.
• Novel Curation Pipeline: A methodology combining a domain-specific AI agent with human-AI collaboration to synthesize missing tests and problem statements, overcoming the "test scarcity" problem in MPC repositories.
• Security-Aware Verification: A dual-stream evaluation framework that integrates dynamic differential testing and static analysis to enforce cryptographic safety and numerical fidelity, criteria absent in general-purpose benchmarks.

4. Results

The authors evaluated eight leading LLMs (including Claude 3.5/4, GPT-4/5, and Gemini 2.5) on the benchmark:

• Functional Resolution: Even the strongest model (Sonnet 4.6) functionally resolved only 22.9% of the 205 tasks. Performance varied significantly by framework; PyTorch-style libraries (CrypTen) were more tractable, while protocol-level systems (MP-SPDZ) remained nearly unsolved.
• Verification Impact: Applying the MPC Verifier significantly reduced the resolution rates. The verified resolution for the top model dropped to 17.1%.
• Rejection of Functional Patches: Up to 40% of patches that passed functional tests were rejected by the verifier due to cryptographic or numerical-fidelity violations.
• Leaderboard Reordering: The inclusion of security verification altered the model ranking; for example, Opus 4.6 overtook Sonnet 4.6 in the verified leaderboard despite lower functional resolution, demonstrating that functional correctness and security safety are distinct signals.

5. Significance and Claims

The paper claims that functional correctness alone substantially overestimates LLM reliability in privacy-preserving software. The results demonstrate that MPC repair demands capabilities beyond what general-purpose benchmarks capture. Specifically:

• Standard fail-to-pass evaluation is insufficient for code that must adhere to strict security and numerical-fidelity guarantees.
• A non-trivial fraction of patches that pass standard tests fail security or numerical checks, indicating that current LLMs struggle with the specific constraints of MPC engineering.
• The methodology serves as a template for benchmarking LLMs in domains where correctness is determined by more than just test outcomes, particularly in security-sensitive software.

The authors note limitations, including the current coverage of only five frameworks, and suggest future work extending the benchmark to other cryptographic stacks like fully homomorphic encryption (FHE) hybrids and zero-knowledge environments. They also emphasize that their curation pipeline is repository-agnostic and reusable for constructing benchmarks in other specialized domains.