Technical Summary: RecourseBench

Problem Statement

The field of algorithmic recourse has seen rapid methodological growth, providing counterfactual explanations to help individuals overturn unfavorable model decisions in high-stakes domains like healthcare and finance. However, the community lacks a principled mechanism for comparing these methods. Existing evaluation frameworks suffer from three primary deficiencies:

1. Lack of Modularity and Interoperability: Prior tools (e.g., Alibi, RobustX) often treat recourse as a sub-feature or rely on tightly coupled architectures (e.g., CARLA) or specific hardware ecosystems (e.g., ReLax/JAX), making it difficult to integrate new methods, datasets, or evaluation metrics without significant refactoring.
2. The Reproducibility Gap: While many methods are published, there is no systematic verification that their implementations faithfully reproduce the originally reported results. Factors such as missing hyperparameters, undocumented configurations, and computational constraints often render independent verification impossible.
3. Context-Dependent Evaluation: Recourse methods optimize for conflicting objectives (proximity, sparsity, robustness, feasibility), and evaluation criteria vary by context (e.g., data shift vs. model multiplicity). Current benchmarks lack the flexibility to accommodate these evolving metrics while maintaining consistent experimental conditions.

Methodology and Framework Design

The authors introduce RecourseBench, a unified evaluation framework built on three core commitments: modularity, reproducibility, and interactivity.

1. Modular Architecture

The framework decomposes the benchmarking pipeline into five fully decoupled layers, governed by abstract interfaces and a dynamic registry:

• Data: Handles raw datasets, train/test splits, and factual sets.
• Preprocessing: Manages feature transformation and normalization.
• Model: Supports target predictive models (e.g., PyTorch, Scikit-learn).
• Recourse Method: The core module where specific algorithms are implemented.
• Evaluation: Computes performance metrics.

Data flows between these layers via DatasetObject containers, ensuring self-contained data transfer and preventing abstraction leaks. The system enforces strict privilege controls (e.g., read-only access for models) to prevent unintended side effects and ensures experimental integrity through a centralized YAML-driven workflow.

2. Reproducibility Protocol

To address the lack of verification in prior benchmarks, RecourseBench introduces a four-tier classification system based on quantitative testing:

• Metric Definition: A result is defined as a tuple (dataset, model, metric). The framework calculates the symmetric relative error ($\Delta$) between the original paper's reported value ($m$) and the framework's implementation ($\hat{m}$).
• Threshold: A result is considered reproduced if $\Delta \leq 0.35$ (35% relative deviation).
• Classification Levels:
  • Level 0: No results reproduced.
  • Level 1: Minimal reproduction (1 result).
  • Level 2: Partial reproduction (some, but not all, results).
  • Level 3: Full reproduction (all reported results).
• Automation: An automated test suite validates every integrated method against its original specifications. Only methods achieving Level 1 or higher are included in the main benchmark evaluations.

3. Evaluation Metrics

The framework evaluates methods across five dimensions using a held-out test set of unfavorable instances:

• Validity: Success rate of achieving the desired class.
• Proximity: Mean Euclidean distance ($\ell_2$) between factual and counterfactual instances.
• Sparsity: Mean number of features modified ($\ell_0$).
• Plausibility: Mean distance to the $Y$-nearest neighbors in the training set ($Y$-NN).
• Runtime: Total generation time.

Scores are normalized to a [0, 1] scale (using negative exponential transformation for "lower-is-better" metrics) and aggregated via a user-defined weighted average, allowing practitioners to prioritize specific dimensions (e.g., speed vs. quality).

4. Interactive Interface

A web-based interface allows users to configure experiments by selecting datasets, model architectures, and specific recourse methods. The system automatically filters incompatible combinations and presents a ranked leaderboard based on the selected metric aggregation strategy.

Key Contributions

1. Unified Modular Framework: RecourseBench is the first framework to strictly decouple data, preprocessing, models, methods, and evaluation, enabling seamless integration of new components without modifying existing code.
2. Enforced Reproducibility: It constitutes the first recourse benchmark to explicitly enforce method-level reproducibility through automated, quantitative testing against original papers, categorizing methods by their fidelity to reported results.
3. Comprehensive Integration: The framework currently integrates 28 state-of-the-art recourse methods spanning six categories: Constraint & Logic-Based, Gradient-Based, Representation Learning, Surrogate-Assisted, Derivative-Free/Heuristic, and Instance-Based Search.
4. Interactive Benchmarking: It provides a configurable interface for multi-metric comparison, allowing users to construct custom leaderboards tailored to specific application constraints.

Results and Empirical Landscape

The authors evaluated 22 methods (those achieving Level 1 or higher) across 3 datasets and 3 model types, resulting in 137 compatible configurations.

• Reproducibility Findings: The distribution of reproducibility levels (Level 0 to 3) reveals a structural infrastructure gap in the field. Many methods suffer from missing artifacts (code, hyperparameters) or computational constraints (e.g., MACE requiring >70 hours per configuration) that prevent faithful reproduction.
• Significance of Level 1: The transition from Level 0 (no reproduction) to Level 1 (minimal reproduction) is identified as the most epistemically significant leap, moving from zero verifiable evidence to independent validation of at least one metric.
• Benchmark Utility: The framework successfully demonstrates that methods optimized for different objectives (e.g., sparsity vs. proximity) yield distinct rankings depending on the user's metric weighting, highlighting the necessity of context-aware evaluation.

Significance and Claims

The paper positions RecourseBench not as a complete reproduction of all existing literature, but as the operationalization of reproducibility. By transforming reproducibility from an implicit expectation into a standardized, measurable property, the framework aims to:

• Provide a reliable basis for comparative evaluation where the fidelity of constituent methods is established.
• Lower barriers for both the reproduction of prior work and the development of new approaches.
• Serve as a shared reference for practitioners to select methods aligned with real-world deployment constraints.

The authors acknowledge that some reproducibility challenges stem from documentation gaps in the literature and that the current framework balances practical feasibility with rigor (e.g., the 35% tolerance threshold and lack of run-level variance aggregation). Future work is invited to address these design choices and expand the method registry.