⚡ electrical engineering

Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

This paper proposes a novel supervised fine-tuning framework that enforces per-sample constraints through a learned, sample-dependent relaxation and an augmented Lagrangian optimization approach, effectively reducing tail constraint violations across diverse tasks like safety, preferences, and length control while preserving overall model performance.

Original authors: Ignacio Hounie, Ignacio Boero, Alejandro Ribeiro

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ignacio Hounie, Ignacio Boero, Alejandro Ribeiro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're training a robot chef to make the perfect sandwich. Your goal is twofold: make it taste delicious (the main objective) and ensure it never uses poison (the constraint).

For a long time, the standard way to teach this robot was to look at the average of all the sandwiches it made. If the robot made 99 safe sandwiches and 1 very poisonous one, the "average" might still look safe enough to pass the test. The problem? That one poisonous sandwich could be the only one a specific customer ever gets, and that's a disaster.

This paper, titled "Every Sample Counts," argues that we need to stop looking at the average and start checking every single sandwich (every single input) to make sure none of them are poisonous.

The Problem with "Average" Safety

The authors show that current methods are like a teacher who only checks the class's final average grade. If the class average is an A, the teacher is happy, even if one student failed the test completely. In the world of AI, this means models might be "safe on average" but still fail spectacularly on difficult or rare requests (the "tail" of the data).

The paper explicitly argues against relying on these average-based constraints or simple fixed penalties (like a "one-size-fits-all" fine). They found that even if you make the average rule stricter, the robot still occasionally makes huge mistakes on the hardest tasks. It's like tightening the speed limit for the whole highway; the fast cars might slow down a bit, but the reckless drivers on the back roads still speed.

The New Solution: A Personalized Coach

The authors propose a new framework called Pointwise Constrained Fine-Tuning. Think of this as giving the robot chef a personal coach who watches every single sandwich as it's being made.

The "Every Sample" Rule: Instead of just checking the average, the coach ensures that every single sandwich meets the safety threshold. If the robot tries to use poison on even one sandwich, the coach stops it immediately.
The Augmented Lagrangian (The Smart Penalty): To make this math work without breaking the robot's brain, they use a clever trick called an "Augmented Lagrangian." Imagine the coach doesn't just yell "No!" but dynamically adjusts the difficulty of the training. If a sandwich is easy to make safe, the coach is relaxed. If a sandwich is tricky, the coach tightens the rules just for that specific moment. This ensures the robot learns to handle the hard cases without forgetting how to make the easy ones.
The "Relaxation" Safety Valve: Sometimes, a request is so weird or impossible that following the rule perfectly would ruin the sandwich entirely (like being asked to make a sandwich out of air). The authors introduce a "learned relaxation." This is like a smart safety valve: the coach allows a tiny, calculated exception only if it's absolutely necessary, but charges a "cost" for doing so. This way, the robot doesn't crash, but it still tries its best to follow the rules.

What They Found (The Evidence)

The team tested this idea on three different "kitchens" (tasks):

Tool Calling: Teaching a robot when to use a tool and when to say "I can't do that."
Safety: Making sure the robot refuses harmful requests without refusing harmless ones.
Re-ranking: Sorting search results to be shorter and faster without losing quality.

In their experiments, which involved models with less than 7 billion parameters (small enough to run on a single powerful graphics card), they measured the results carefully.

The "Tail" Problem Solved: When they looked at the distribution of mistakes, their method (the "pointwise" approach) had almost zero large violations. In contrast, the old "average" methods still had a "long tail" of big mistakes. For example, in the safety tests, their method achieved a 100% refusal rate on harmful prompts while only reducing the helpfulness score by about 5%. The old methods, to get the same safety, cut helpfulness by 9%.
Better Trade-offs: They showed that their method creates a better balance (a "Pareto frontier"). It's like getting a faster car that also uses less gas, whereas the old methods forced you to choose between speed and fuel efficiency.
No Magic, Just Math: The authors are careful to note that this isn't a magic wand that solves everything. They measured these results on specific datasets (like the "When2call" dataset with 4.5k examples and the "MS MARCO" dataset with 50k training instances). They found that while the method works great for these specific tasks, the theoretical guarantees for all possible scenarios are still being worked out because neural networks are complex and not perfectly predictable.

The Bottom Line

The paper suggests that if you want an AI that is truly reliable, you can't just look at the average. You have to enforce the rules on every single sample. By using a smart, dynamic system that adjusts penalties for each specific input, you can stop the "rare but dangerous" failures without making the AI worse at its main job. It's a shift from "good enough on average" to "safe every single time."

Technical Summary: Every Sample Counts – Supervised Fine-Tuning of Language Models with Pointwise Constraints

1. Problem Statement

Current methods for aligning and fine-tuning Large Language Models (LMs) typically rely on constrained optimization frameworks that enforce requirements (such as safety, length control, or tool-calling preferences) only in expectation over the input distribution. While these methods ensure that constraints are satisfied on average, they suffer from a critical limitation: they do not guarantee constraint satisfaction for individual inputs.

This "average-only" approach leads to undesirable disparities, particularly for tail data (inputs that are underrepresented in training or inherently more difficult). Empirical evidence suggests that models satisfying constraints on average may still exhibit significant violations on critical subsets of inputs. Existing methods often fail to distinguish between "easy" and "hard" samples, applying uniform penalties that either over-penalize simple cases or fail to constrain difficult ones sufficiently.

The paper proposes a shift from average-constrained learning to pointwise constrained fine-tuning, where constraints must hold for almost every individual sample in the input distribution, rather than just in expectation.

2. Methodology

The proposed framework addresses the challenges of enforcing pointwise constraints in non-convex neural network optimization through three core components:

A. Pointwise Constrained Formulation

The authors formulate the fine-tuning problem as minimizing an expected objective loss $\ell_0$ subject to inequality constraints $\ell_i \leq \varepsilon$ holding pointwise (almost everywhere) over the data distribution $D$ .
$\pi^*_\theta = \arg \min_{\theta} \mathbb{E}_{(x,y)\sim D} [\ell_0] \quad \text{s.t.} \quad \ell_i(\pi_\theta, x, y) \leq \varepsilon_i, \quad D\text{-a.e.}$
Unlike average constraints which use a single scalar multiplier, this formulation requires a distinct multiplier for every sample, allowing the penalty to adapt to the specific difficulty of each input.

B. Augmented Lagrangian Approach

To solve this non-convex problem, the authors utilize the dual domain. However, standard Lagrangian duality offers weak guarantees for non-convex problems (like deep neural networks) due to the duality gap. To overcome this, the paper employs Augmented Lagrangian (AL) methods.

The AL method replaces the linear dependence between the Lagrange multiplier $\lambda$ and the constraint violation with a quadratic term.
This provides additional curvature correction, enabling strong duality and primal recoverability in a wider range of non-convex settings compared to standard Lagrangian approaches.
The optimization is performed via Empirical Dual Gradient Ascent, where the functional multiplier $\lambda(x,y)$ is approximated by sample-wise multipliers $\lambda_i$ for a finite dataset.

C. Learned Pointwise Relaxation

Pointwise constraints can be overly restrictive, especially in the presence of outliers or when constraints conflict with the primary objective. To mitigate this, the authors introduce a learned, sample-dependent relaxation.

A relaxation function $u(x,y)$ is introduced to increase the admissible threshold from $\varepsilon$ to $\varepsilon + u(x,y)$ .
A convex cost function $c(u)$ (specifically a quadratic penalty $\beta \|u\|^2$ ) is added to the objective to penalize the severity and frequency of these relaxations.
Crucially, the relaxation variable $u$ does not need to be optimized directly. Through duality, its effect is absorbed into the dual problem as a regularization term and a rescaling of the dual variables. This allows the model to trade off a user-defined relaxation cost against the training objective, effectively controlling the "tail" of the constraint violation distribution.

3. Key Contributions

Pointwise Constraint Framework: The paper proposes a novel alignment framework that enforces constraints on individual inputs rather than in expectation, addressing the disparity in performance across the input distribution.
Augmented Lagrangian for Non-Convex Learning: The authors apply Augmented Lagrangian methods to pointwise constrained fine-tuning, overcoming the lack of strong duality in non-convex neural network optimization.
Learned Relaxation Mechanism: A mechanism is introduced to jointly solve the learning task and the relaxation of constraints, minimizing a user-defined cost to handle outliers and conflicting objectives without manual tuning of per-sample thresholds.
Empirical Validation: The framework is instantiated across three distinct small language model (SLM) fine-tuning tasks:
- Instruction Following & Safety: Enforcing refusal of harmful prompts while maintaining helpfulness on safe prompts.
- Tool Calling Preferences: Enforcing correct behavior selection (e.g., calling a tool vs. hallucinating) with length-normalized log-probability constraints.
- Re-ranking & Length Optimization: Balancing retrieval relevance with passage length (token efficiency).

4. Results

The authors empirically validate that their approach reduces tail constraint violations while largely preserving downstream performance:

Reduction of Tail Violations: Across all tasks, the proposed method significantly reduces the frequency and severity of large constraint violations compared to average-constrained and fixed-penalty baselines. The Cumulative Distribution Function (CDF) of constraint violations shows that while average methods satisfy "easy" samples with large margins, they fail on "hard" samples; the pointwise approach ensures consistent performance.
Improved Trade-offs: The method achieves better Pareto frontiers between the objective (e.g., helpfulness, relevance) and constraint metrics.
- In Safety, the pointwise method achieved a 100% refusal rate on unsafe prompts with only a 5% reduction in helpfulness (win rate), whereas average constraints required a 9% reduction in helpfulness to achieve similar safety.
- In Tool Calling, pointwise constraints yielded a more uniform improvement across error categories compared to methods that skewed performance toward specific categories.
- In Re-ranking, the method successfully reduced passage length without degrading relevance metrics (MRR@10), a trade-off that average constraints failed to achieve without sacrificing quality.
Relaxation Control: The hyperparameter $\beta$ in the relaxation cost effectively controls the trade-off between strict constraint adherence and objective performance, allowing users to tune the model's behavior based on specific risk tolerances.

5. Significance and Claims

The paper claims that enforcing constraints per-sample is a necessary evolution for deploying LMs in high-stakes domains where average performance is insufficient. The significance of the work lies in:

Addressing Distributional Disparities: It directly tackles the issue where models underperform on tail data, ensuring that critical constraints (like safety) are not violated for specific, potentially vulnerable, inputs.
Practical Optimization: By leveraging Augmented Lagrangian methods and learned relaxation, the framework provides a practical, scalable solution to pointwise constraints that avoids the intractability of enforcing hard constraints on every sample without relaxation.
Flexibility: The framework is shown to be adaptable to diverse constraints (safety, preferences, length) and tasks, offering a unified approach to constrained fine-tuning that outperforms existing average-constrained and fixed-penalty methods in reducing worst-case violations.

The authors conclude that while average constraints may suffice for general performance, pointwise constraints are essential for robust, equitable, and safe deployment, offering superior trade-offs between utility and constraint satisfaction.