Technical Summary: Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data

1. Problem Statement

Safe Reinforcement Learning (RL) aims to derive policies that satisfy predefined safety constraints, which is critical for applications like autonomous driving and robotics. While offline safe RL offers a paradigm to learn from pre-collected datasets without risky online exploration, existing methods face a fundamental limitation in high-stakes scenarios: the scarcity or total absence of unsafe samples.

In many practical data collection processes (e.g., robot manipulation), external interventions often prevent agents from reaching unsafe states, resulting in datasets composed almost entirely of safe trajectories. Conventional offline safe RL methods, which rely on learning cost value functions from abundant unsafe samples to define safety boundaries, fail in this setting. They tend to treat all observed data as uniformly safe, overlooking "safe-but-infeasible" states—states that currently satisfy constraints but inevitably violate them within a few steps due to dynamics (e.g., inertia). This oversight leads to policy deployment failures where agents drift into unsafe regions.

The core challenge is: How can we learn a safe policy offline when unsafe samples are scarce or entirely absent?

2. Methodology: PROCO

The authors propose PROCO (Model-Based Proactive Cost Generation), a model-based offline safe RL framework designed to identify infeasible states and learn safe policies from datasets with few or no unsafe samples. The methodology integrates three key components:

A. LLM-Assisted Conservative Cost Function Generation

To bridge the gap caused by the lack of unsafe data, PROCO leverages Large Language Models (LLMs) to incorporate natural language knowledge into the learning process.

• Input: The LLM receives task information ($L_{task}$), a natural language description of safety constraints ($L_{cost}$), and explicit instructions ($L_{inst}$) to generate a cost function that is more conservative than the strict constraint definition.
• Output: A Python function $\bar{c}$ that labels states as unsafe (1) or safe (0).
• Validation and Feedback Loop: Since LLM outputs may be unreliable, PROCO employs a validation mechanism:
  1. Safety Check: The generated cost function is validated against a small set of known unsafe samples ($D_{unsafe}$, $\le 100$ samples). It must achieve 100% accuracy on these to ensure no known unsafe states are missed.
  2. Conservativeness Check: The function is evaluated on the safe dataset ($D$). The proportion of safe samples labeled as unsafe is measured. If this proportion falls within a hyperparameter-controlled range $[p_{min}, p_{max}]$, the function is accepted.
  3. Refinement: If the function is too conservative or not conservative enough, a feedback description ($L_{feed}$) is generated and fed back to the LLM to refine the cost function.

B. Dynamics Model-Based Feasibility Identification

PROCO learns an ensemble dynamics model $\hat{T}$ from the offline data. To identify infeasible states without observed violations, it performs branched model rollouts:

• Starting from offline data samples, the model simulates future trajectories.
• A conservative feasible Bellman operator ($\bar{B}^*$) is introduced. Unlike standard operators, $\bar{B}^*$ considers the worst-case transition within the ensemble of dynamics models to ensure robustness against model uncertainty.
• This process proactively generates diverse counterfactual unsafe samples. By labeling states near actual unsafe states as unsafe (via the conservative cost function), the method effectively shortens the transition steps from infeasible to unsafe states, reducing the impact of model errors on feasibility identification.

C. Policy Learning Pipeline

1. Data Augmentation: The learned dynamics model $\hat{T}$ and the conservative cost function $\bar{c}$ are used to perform model rollouts. Trajectories containing safety violations (according to $\bar{c}$) are added to a synthetic dataset $D_{\hat{T}}$.
2. Relabeling: The offline dataset $D$ is re-labeled with constraint violation costs using $\bar{c}$.
3. Value Function Update: The algorithm updates the constraint violation value function ($V_h$) and action-value function ($Q_h$) using the combined dataset ($D \cup D_{\hat{T}}$). It employs reverse expectile regression to approximate the minimization operator in the feasible Bellman update.
4. Policy Optimization: The policy is updated using the original offline dataset $D$, guided by the cost critics learned from the augmented data.

3. Key Contributions

1. Problem Formulation: The paper identifies and formalizes the challenge of learning safe policies from "safe-only" datasets where infeasible states are indistinguishable from safe ones without unsafe samples.
2. Knowledge-Data Integration: It introduces a novel framework (PROCO) that integrates natural language safety specifications (via LLMs) with empirical offline data to generate conservative cost functions, enabling risk estimation without observed violations.
3. Proactive Feasibility Identification: By combining a learned dynamics model with a conservative cost function, PROCO synthesizes counterfactual unsafe samples to identify infeasible states, addressing the "safe-but-infeasible" blind spot of existing methods.
4. Theoretical Guarantees: The authors provide theoretical analysis (Theorem 4.8) demonstrating that integrating PROCO with any baseline offline safe RL algorithm does not degrade safety performance due to the underestimation of safety values, even without assumptions on model error magnitude.

4. Experimental Results

The authors evaluated PROCO on 17 tasks from the Safety-Gymnasium benchmark (Navigation and Velocity tasks) using the OSRL dataset, specifically constructing "safe-only" datasets by removing all unsafe data.

• Performance: PROCO significantly outperformed original offline safe RL algorithms (FISOR, LSPC, CAPS) and behavior cloning baselines (BC, CDT) on safe-only datasets.
  • It achieved over 400% improvement in safety performance across various algorithms, with some tasks showing improvements exceeding 1000%.
  • In the "Ant Circle" case study, PROCO successfully identified infeasible states near the safety boundary and achieved zero safety violations, whereas baselines failed to distinguish feasible from infeasible states and incurred significant violations.
• Robustness: The method remained effective even when the amount of available unsafe data was extremely limited (e.g., 10% of the original unsafe dataset).
• Ablation Studies:
  • Removing the dynamics model or the conservative cost function generation led to significant drops in safety performance.
  • The feedback mechanism for the LLM-generated cost function was crucial for achieving the desired level of conservativeness.
  • The method showed stability across different LLMs (GPT-o4-mini, Gemini 2.5 Pro), though performance varied slightly.

5. Significance and Claims

The paper claims that PROCO addresses a critical bottleneck in deploying safe RL in real-world, high-stakes environments where collecting unsafe data is infeasible or dangerous. By leveraging LLMs to translate natural language safety constraints into conservative cost functions and using model rollouts to proactively explore potential failures, PROCO enables the learning of safe policies from datasets that would otherwise be insufficient for safety-critical decision-making.

The authors emphasize that their approach does not require unsafe samples to learn a safe policy, making it suitable for scenarios where data is collected via human teleoperation or external intervention that prevents collisions. They conclude that while the method currently relies on state-based observations and LLMs, it represents a significant step toward knowledge-driven offline safe RL. Future work is suggested to extend this to visual tasks using Vision-Language Models (VLMs).