Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference

1. Problem Statement

Neural Natural Language Inference (NLI) models, particularly those based on pre-trained transformers (e.g., BERT, ELECTRA), often achieve high benchmark accuracy by exploiting dataset artifacts rather than performing genuine semantic reasoning.

• The Artifact Problem: Models learn spurious correlations between specific lexical patterns in the hypothesis and the target label, ignoring the premise.
• Evidence: A "hypothesis-only" model (which receives only the hypothesis and no premise information) achieves 57.7% accuracy on the SNLI dataset, significantly above the 33.3% random baseline.
• Consequence: Approximately 38.6% of errors made by standard baseline models are driven by these artifacts. This reliance limits model robustness, causing failures under distribution shifts, adversarial examples, or when surface patterns do not align with semantic truth (e.g., negation or numerical reasoning).

2. Methodology: Product-of-Experts (PoE) Training

The paper proposes Product-of-Experts (PoE) training, a debiasing mechanism that dynamically reweights training examples based on the confidence of a "bias model."

Core Mechanism

Instead of treating all training examples equally, PoE downweights examples where a bias model is highly confident. The intuition is that if a bias model (trained only on the hypothesis) is confident, the example likely relies on spurious artifacts rather than genuine premise-hypothesis reasoning.

Technical Implementation

1. Bias Model: A separate model (using the same architecture, e.g., ELECTRA-small) is trained or utilized to predict labels using only the hypothesis text ($h$).
2. Confidence Calculation: For each training instance $x_i$, the bias model $B$ computes a confidence score:
   $$\text{confidence}(B(x_i)) = \max_c P(y=c \mid \text{hypothesis-only})$$
3. Weighted Loss: The standard cross-entropy loss is modified by an example-specific weight $w_i$:
   $$w_i = \frac{1}{\text{confidence}(B(x_i))^\lambda + \epsilon}$$
   • $\lambda$ (Debiasing Strength): A hyperparameter controlling the aggressiveness of downweighting.
   • $\epsilon$: A small constant for numerical stability.
4. Optimization: The final loss $L$ is computed as a weighted average:
   $$L = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{w_i}{\bar{w}} \right) \cdot \ell(y_i, \hat{y}_i)$$
   where $\bar{w}$ normalizes the weights to maintain gradient scale.

Key Design Choices

• Architecture: The main model architecture remains identical to standard training (ELECTRA-small-discriminator); only the loss function changes.
• Hyperparameter Tuning: An ablation study identified $\lambda = 1.5$ as the optimal value, balancing accuracy preservation with bias reduction.
• Efficiency: The method adds negligible computational overhead because the bias model requires only a single forward pass per example, and the main model's architecture is unchanged.

3. Key Contributions

1. Quantification of Artifacts: Demonstrated that hypothesis-only models achieve 57.7% accuracy on SNLI, proving that nearly 40% of baseline errors are artifact-driven.
2. PoE Training Framework: Introduced a dynamic reweighting scheme that reduces reliance on artifacts without altering model architecture.
3. Ablation and Analysis: Identified $\lambda = 1.5$ as the optimal trade-off point and provided detailed analysis showing that PoE shifts model attention toward examples requiring genuine premise-hypothesis interaction.
4. Behavioral Evaluation: Conducted rigorous testing on negation, paraphrase, and lexical overlap invariance, showing that PoE models are more robust to adversarial perturbations than standard models.

4. Experimental Results

Experiments were conducted on the SNLI dataset (570K pairs) using an ELECTRA-small model on an NVIDIA RTX 3090.

• Accuracy Preservation: PoE maintains near-baseline accuracy (only a 0.20% drop), proving that debiasing does not require sacrificing performance.
• Bias Reduction: The model's agreement with the hypothesis-only bias model dropped by 4.85 percentage points, indicating a significant reduction in artifact reliance.
• Robustness: On behavioral tests (negation sensitivity, lexical overlap invariance), PoE models outperformed standard models. For example, on a subset of negation-heavy examples, PoE achieved 82.7% accuracy vs. 78.9% for standard training.
• Limitations: Both models still struggle with complex reasoning tasks like numerical reasoning and double negation, suggesting that PoE reduces artifact reliance but does not fully solve deep semantic reasoning challenges.

5. Significance and Implications

• Efficiency vs. Robustness: The paper demonstrates that robustness can be achieved with negligible computational cost (approx. 3 hours on a single GPU) and without architectural complexity.
• Training Objective Importance: It highlights that the choice of training objective (standard vs. PoE) is more critical than model architecture for ensuring generalization in NLI.
• Real-World Applicability: In high-stakes domains (legal, medical), where robustness and fairness are paramount, PoE offers a practical path to reduce spurious correlations without significant performance penalties.
• Future Directions: The authors suggest that while PoE is a strong step forward, future work should combine it with syntactic modeling, curriculum learning, or adaptive $\lambda$ scheduling to address deeper reasoning gaps like numerical inference.

In conclusion, the paper establishes that Product-of-Experts training is an effective, low-cost strategy to mitigate dataset artifacts in NLI, enabling models to rely more on genuine semantic reasoning while maintaining competitive benchmark scores.