=== SUMMARY ===

Technical Summary: From LLM-Generated Specifications to Learned Quadruped Locomotion

Problem Statement

Deep reinforcement learning (RL) has enabled quadruped robots to learn agile locomotion, yet performance remains heavily dependent on the manual engineering of reward functions. Designing these rewards requires substantial domain expertise to balance local terms (tracking, posture, energy) and often relies on empirical tuning rather than first principles. Furthermore, numerical local rewards fail to explicitly describe the desired global temporal behavior, which is particularly critical for multi-gait locomotion where walking, trotting, and bounding differ in contact timing and support patterns. While formal specifications like Signal Temporal Logic (STL) offer interpretability and quantitative robustness, writing them manually still requires significant expertise. Conversely, recent Large Language Model (LLM) approaches that generate reward code directly from natural language often lack the structural rigor needed for complex temporal constraints. This paper addresses the gap of generating interpretable, temporally structured reward specifications using LLMs while grounding their numerical parameters in expert data.

Methodology

The authors propose a pipeline that leverages LLMs to generate the structure of Parametric Signal Temporal Logic (PSTL) specifications, while using expert trajectories to instantiate the parameters and filter the output.

1. LLM Specification Generation:

   • Models (GPT-5.5 and Qwen 3.6) are prompted with natural language locomotion objectives and a constrained STL grammar.
   • Crucially, the LLMs are asked to propose only the symbolic structure (templates) for command tracking, safety, and gait structure. Numerical thresholds and temporal constants are left as symbolic parameters to be estimated later, preventing the LLM from inventing arbitrary values.
   • Two settings are explored:
     • Gait-Aware (Multi-Gait): The prompt defines three speed regimes (walking-trot, trot, bound) based on Froude number transitions. The LLM generates distinct specifications for each regime.
     • Gait-Agnostic: The prompt does not prescribe specific gaits, allowing the robot to discover contact patterns.

2. Data Grounding and Expert-Consistency Filtering:

   • Numerical parameters for the generated PSTL templates are estimated from a dataset of 50 expert trajectories per regime.
   • A filtering mechanism is applied: a generated specification is retained only if its median robustness over the expert trajectories is non-negative ($Q_{0.50}(R_\phi) \ge 0$). This discards specifications that are systematically violated by expert behavior, ensuring the reward signal aligns with demonstrated competence.

3. Reward Construction and Training:

   • The retained specifications are converted into smooth, finite-history reward functions using STL robustness semantics.
   • Robustness values for active specifications are aggregated using a soft-min function and normalized via $\tanh$ to prevent dominance by large magnitudes.
   • The final scalar reward is a weighted sum of safety, tracking, and pattern terms.
   • Policies are trained using Proximal Policy Optimization (PPO) in the MuJoCo XLA (MJX) simulation environment with the Barkour quadruped robot.

Key Contributions

1. LLM-Expert Hybrid Pipeline: A novel framework where LLMs generate the symbolic structure of interpretable locomotion specifications, while expert data grounds the numerical parameters.
2. Expert-Consistency Filter: A mechanism to discard LLM-generated specifications that contradict demonstrated expert behavior (median robustness < 0), preventing the introduction of systematically violated constraints into the reward.
3. Comparative Evaluation of Formulations: An investigation into how explicitly prescribing gait structure (gait-aware) versus allowing emergent behavior (gait-agnostic) affects learned locomotion.
4. Benchmarking: A comprehensive comparison against hand-engineered heuristics, direct LLM-generated reward code (Text2Reward), and an expert-switching oracle.

Results

The study evaluates performance across forward speeds from 0.3 m/s to 2.1 m/s using metrics including Cost of Transportation (CoT), survival rate, command success, and gait match.

• Gait-Agnostic Performance:

  • GPT-5.5 and Text2Reward (Gait-Agnostic) achieved the strongest quantitative performance, maintaining 100% survival and command success across all speeds with low CoT.
  • However, visual inspection revealed a critical flaw: these policies learned a "bound-like" contact pattern across the entire speed range, including low speeds (0.3 m/s). This resulted in unnatural, high-cadence leg movements and vertical oscillations, indicating that high quantitative success scores do not guarantee dynamically appropriate control.
  • Qwen 3.6 (Gait-Agnostic) failed to survive at speeds $\ge 1.6$ m/s.

• Gait-Aware (Multi-Gait) Performance:

  • Qwen 3.6 (Multi-Gait): Achieved 100% survival and command success across the full speed range (0.3–2.1 m/s) and successfully matched the target "bound" gait at high speeds.
  • GPT-5.5 (Multi-Gait): Captured low/mid-speed gaits well but failed command tracking at speeds $\ge 1.9$ m/s.
  • Text2Reward (Multi-Gait): Failed completely at high speeds (1.9–2.1 m/s), terminating in every rollout.
  • Hand-Engineered (Heuristic): Lost tracking accuracy at the highest speeds (2.0–2.1 m/s).

• Ablation on Robustness Horizon ($H$):

  • Shorter temporal horizons ($H \in \{1, 5\}$) generally yielded more stable and successful policies.
  • Longer horizons ($H=20, 30$) often degraded performance, as the "always" ($G$) operator in STL depends on the worst value in the window, keeping past violations in the reward signal longer and complicating credit assignment.

• Model Comparison:

  • The relative performance of GPT-5.5 and Qwen 3.6 was highly dependent on the control setting. GPT-5.5 excelled in the gait-agnostic setting, while Qwen 3.6 outperformed it in the multi-gait setting, particularly at high speeds.

Significance and Claims

The paper claims that inserting temporal logic as an intermediate representation between LLM generation and policy learning offers distinct advantages over direct reward code generation, particularly for high-speed locomotion. The STL-based approach (specifically Qwen 3.6 in the multi-gait setting) successfully learned the desired high-speed "bound" gait where direct code generation (Text2Reward) failed.

However, the authors maintain a modest stance regarding the findings:

• No Universal Dominance: No single reward formulation universally dominates across both task settings (gait-aware vs. agnostic) and all evaluation criteria.
• Limitations of Quantitative Metrics: The results highlight that high command-tracking success does not guarantee the recovery of intended gait structures or natural motion, as seen in the gait-agnostic policies that adopted a bound gait at low speeds.
• Simulation Constraint: The authors explicitly note that evaluations are confined to simulation (MJX). While domain randomization was used, the sim-to-real transferability of these learned multi-gait behaviors has not yet been established on physical hardware.

The work demonstrates that LLMs can effectively propose the structure of formal specifications, but the parameters and validity of these specifications must be rigorously grounded in expert data to produce robust, interpretable, and effective locomotion policies.