Technical Summary: When Bits Break Recourse: Counterfactual-Faithful Quantization

1. Problem Statement

Modern machine learning systems increasingly rely on quantization to reduce memory footprints and inference latency, particularly in mobile and edge deployments. While Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ) are rigorously evaluated for their ability to preserve predictive accuracy, this paper identifies a critical deployment mismatch: quantization can preserve average-case accuracy while silently breaking algorithmic recourse.

Algorithmic recourse provides actionable recommendations (minimal changes to an input) that flip a model's decision to a desired outcome. The paper argues that accuracy constraints apply to the data distribution, whereas recourse depends on the local geometry of the decision boundary in specific neighborhoods around individual points. Quantization perturbs weights and activations in a structured but non-uniform manner, which can shift decision margins and rotate boundaries precisely in the neighborhoods that matter for recourse. Consequently, a recourse action valid for a full-precision model may become invalid or significantly more costly after quantization, even if the model's test accuracy remains unchanged.

2. Methodology: Counterfactual-Faithful Quantization (CFQ)

The authors propose Counterfactual-Faithful Quantization (CFQ), a quantization-aware training method designed to preserve counterfactual behavior as a first-class objective alongside predictive accuracy.

Core Mechanism

CFQ treats quantization as a structured model shift. It utilizes a "teacher" full-precision model ($f$) to generate recourse actions ($\delta_{fp}$) for input instances. These actions are computed via a small number ($K$) of projected gradient steps on a differentiable counterfactual loss and are treated as constants (via stop-gradient) during the optimization of the quantized model ($f_q$).

The training objective for CFQ combines three components:

1. Task Loss ($L_{task}$): Standard supervised loss (e.g., cross-entropy) to maintain predictive accuracy on factual inputs.
2. Validity Preservation Loss ($L_{valid}$): A regularizer that forces the quantized model to predict the target outcome at the teacher's recourse point ($x + \delta_{fp}$). This ensures that if an action works for the full-precision model, it remains valid for the quantized model.
3. Bit Cost Penalty ($\lambda \cdot \text{BitCost}$): A constraint enforcing a global bit budget, enabling mixed-precision allocation.

Mixed-Precision Bit Allocation

CFQ jointly optimizes quantizer parameters (e.g., step sizes, clipping thresholds) and layer-wise bitwidths. It employs a differentiable relaxation using Gumbel-Softmax to select bitwidths from a candidate set (e.g., $\{2, 3, 4, 8\}$) for each layer. The optimization is guided by the counterfactual objective, allocating higher precision to layers that most influence the validity of recourse points, rather than solely optimizing for average accuracy.

Optional Enhancements

• Direction/Cost Matching ($L_{match}$): An optional term that aligns the recourse action computed by the quantized model ($\delta_q$) with the teacher action ($\delta_{fp}$) in terms of direction and cost.
• Margin Stabilization ($L_{hinge}$): A hinge loss that enforces a positive margin at the teacher recourse point, providing a safety buffer against quantization-induced perturbations.

Theoretical Foundation

The paper provides a sufficient condition for recourse transfer based on margin robustness. It posits that if the full-precision target margin at a recourse point exceeds twice the maximum logit perturbation induced by quantization ($\epsilon$), the quantized model will preserve the target label. CFQ's objective directly addresses this by increasing the margin at recourse points and reducing effective perturbations through targeted bit allocation.

3. Key Contributions

1. Identification of a Deployment Gap: The authors demonstrate that accuracy-preserving quantization baselines (uniform QAT, mixed-precision QAT) can significantly degrade counterfactual transferability, leading to recourse failures invisible to standard accuracy metrics.
2. New Metrics: Two metrics are introduced to quantify this failure mode:
   • Validity Drop (VD): The probability that a recourse action valid for the full-precision model fails for the quantized model.
   • Counterfactual Recourse Gap (CRG): The relative increase in the minimal cost required to achieve the target outcome after quantization.
3. CFQ Algorithm: A novel training procedure that couples mixed-precision bit allocation with a counterfactual-consistency regularizer to preserve recourse validity under bit budgets.
4. Empirical Validation: Extensive experiments on tabular benchmarks (ADULT, GERMAN CREDIT, COMPAS) and non-tabular image datasets (MNIST, Fashion-MNIST, CelebA) showing that CFQ maintains accuracy while substantially improving VD and CRG.

4. Experimental Results

Experiments were conducted on standard tabular recourse benchmarks (ADULT, GERMAN CREDIT, COMPAS) and extended to image datasets using structured latent or semantic action spaces.

• Accuracy vs. Recourse: Baseline methods (LSQ-QAT, PACT-QAT, accuracy-driven Mixed-Precision) achieved predictive accuracy comparable to full-precision models (e.g., ~85.7% on ADULT) but suffered high Validity Drop (VD $\approx$ 0.12–0.18) and high Counterfactual Recourse Gap (CRG $\approx$ 0.16–0.23).
• CFQ Performance: CFQ achieved comparable accuracy (e.g., 85.7% on ADULT) while drastically reducing VD (to ~0.06) and CRG (to ~0.07). This represents roughly a 50% reduction in recourse failure rates compared to the best non-CFQ baselines.
• Robustness: The benefits of CFQ were consistent across different bit budgets, with the largest gains observed in low-bit regimes (e.g., 3-4 bits) where quantization noise is strongest.
• Training-Free Variant (CF-PTQ): A post-training variant was introduced that calibrates quantizers on both factual and teacher-counterfactual points without updating backbone weights. This variant significantly reduced VD and CRG compared to standard PTQ, offering a practical middle ground between standard PTQ and full QAT.
• Subgroup Analysis: CFQ reduced recourse instability across different subgroups (e.g., gender, race), though subgroup disparities in stability could still exist, suggesting a need for future fairness-aware extensions.

5. Significance and Claims

The paper claims that predictive accuracy is insufficient to guarantee the reliability of algorithmic recourse in compressed models. By treating quantization as a model change that alters local decision geometry, the authors argue that deployment pipelines must explicitly optimize for recourse stability.

The significance of CFQ lies in its ability to:

• Prevent Broken Promises: Ensure that actionable recommendations generated during development (on full-precision models) remain valid and cost-effective after deployment (on quantized models).
• Provide Measurable Stability: Introduce VD and CRG as standard metrics to evaluate the robustness of recourse under compression, revealing failures that accuracy metrics hide.
• Offer Practical Solutions: Provide both a training-based (CFQ) and a training-free (CF-PTQ) approach to achieve this stability, making it applicable to various deployment constraints.

The authors maintain a modest stance, noting that CFQ preserves recourse with respect to the specified predictive model and action set but does not certify causal effectiveness in the real world. They acknowledge that CFQ introduces computational overhead (primarily from teacher recourse computation) and that its stability is calibrated to the training distribution, potentially requiring recalibration under severe deployment shifts. However, the results demonstrate that optimizing quantization for counterfactual behavior is a distinct and necessary objective for high-stakes decision-support systems.