QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Systems

QuickLAP is a Bayesian framework that leverages Large Language Models to fuse ambiguous physical corrections with high-level language feedback, enabling semi-autonomous systems to rapidly and robustly infer user reward functions in real time.

The Gist

Imagine you are teaching a robot how to drive or how to move its arm. You have two ways to tell it what you want: you can do it (physically nudging the steering wheel or grabbing the robot's arm) or you can say it (telling it "Watch out for that cone!").

The problem is that neither method works perfectly on its own:

• Doing it (Physical Correction): If you grab the steering wheel to turn left, the robot knows where to go, but it doesn't know why. Did you turn left to avoid a cone? To change lanes? Or because you saw a puddle? The robot is left guessing.
• Saying it (Language): If you shout "Avoid the cones!", the robot knows what you care about, but it doesn't know exactly how to move to avoid them. It's like being told "Be careful!" without knowing what you are being careful about.

Enter QuickLAP.

Think of QuickLAP as a super-smart translator that sits between you and the robot. It's a new way for robots to learn from you in real-time by combining your actions and your words into one clear instruction.

How It Works: The "Detective" Analogy

Imagine the robot is a detective trying to solve a mystery: "What does my human boss actually want?"

1. The Clue (Physical Action): You nudge the robot's arm. The detective sees the movement. "Okay, the boss moved the arm away from the red block."
2. The Testimony (Language): At the same time, you say, "Don't hit the red block!"
3. The Brain (QuickLAP): QuickLAP uses a special AI (a Large Language Model) to act as the detective's brain. It looks at your words and asks:
   • Which part of the movement matters? (The "Attention" part).
   • How sure are you about this? (The "Confidence" part).
   • How much should I change my plan based on your words?

If you say "Don't hit the red block!" while nudging the arm, QuickLAP realizes: "Ah, the boss is specifically worried about the red block, not the speed or the path. I should focus my learning on avoiding that specific object."

If you just say "Be careful!" while nudging the arm, QuickLAP gets a bit confused. It knows you are worried, but it's not sure what. So, it leans more heavily on the physical nudge to figure out what you actually did, rather than guessing wildly based on vague words.

The Magic Formula

The paper describes a mathematical "recipe" (a Bayesian framework) that mixes these two ingredients:

• The Physical Nudge: Tells the robot the direction of the change.
• The Words: Tell the robot which specific thing to focus on and how strongly to change its mind.

By mixing them, QuickLAP can update the robot's "brain" (its reward function) instantly. It's like if you were teaching a dog to sit. If you push the dog down (physical) and say "Sit!" (language) at the same time, the dog learns instantly. If you just push the dog down without saying anything, the dog might think you want it to lie down. If you just say "Sit!" while the dog is running, the dog is confused. QuickLAP makes sure the robot gets the perfect mix of both.

What They Found

The researchers tested this in two worlds:

1. A Robot Arm: Trying to move a block without hitting obstacles.
2. A Self-Driving Car: Trying to drive around cones and puddles.

The Results:

• Less Mistakes: When using QuickLAP, the robot made over 70% fewer mistakes in learning what you wanted compared to methods that only used physical nudges or simple combinations of words and actions.
• Better Understanding: In tests with real humans, people felt that QuickLAP understood them much better. They felt more collaborative, as if the robot was "listening" to their words to understand their actions.
• Handling Confusion: When humans gave vague instructions (like "Watch out!") or made mistakes (saying "cone" but moving toward a "puddle"), QuickLAP was able to figure out the true intent by looking at the physical action to ground the words.

The Bottom Line

QuickLAP is a system that lets robots learn faster and more accurately by treating your words as a guide to help interpret your actions. It stops the robot from guessing why you moved it and helps it understand exactly what you care about, making human-robot teamwork much smoother and more intuitive.

Technical Summary

Technical Summary: QuickLAP

Problem Statement

Robots operating in semi-autonomous systems must learn human preferences to align their behavior with user intent. Humans typically convey these preferences through two modalities: physical corrections (e.g., nudging a steering wheel or robot arm) and natural language (e.g., "Stay away from the cones"). However, relying on either modality in isolation presents significant limitations:

• Physical Corrections: While grounded in the physical environment and precise in execution, they are often ambiguous regarding intent. A single physical adjustment might simultaneously affect multiple reward features (e.g., avoiding a cone might inadvertently change lane alignment), making it difficult for the robot to infer which specific feature the user intended to modify.
• Natural Language: While capable of expressing high-level goals and clarifying intent, language often lacks physical grounding. Utterances can be vague ("Stay away!"), underspecified, or context-dependent, failing to convey exactly how the robot should act without reference to the current state.

Existing methods often fail to fully exploit the complementarity of these signals. Some rely on large, offline datasets of paired trajectories and language, limiting online adaptation. Others treat language as precise and self-contained, ignoring the physical context necessary to disambiguate vague instructions. There is a need for a framework that can fuse these modalities in real-time to resolve ambiguity and infer reward functions efficiently.

Methodology: QuickLAP

The authors propose QuickLAP (Quick Language–Action Preference learning), a Bayesian framework designed for real-time reward inference from joint physical and language feedback. The core insight is to treat language not merely as a description of goals, but as a probabilistic observation over the user's latent reward preferences. This observation modulates how physical corrections are interpreted.

Framework Overview

QuickLAP computes the posterior distribution over preference parameters $\theta$ given a robot's proposed trajectory $\xi_R$, a human's corrective trajectory $\xi_H$, and a natural language utterance $l$:
$$P(\theta | \xi_H, \xi_R, l) \propto P(\xi_H | \xi_R, \theta) \cdot P(l | \xi_H, \xi_R, \theta) \cdot P(\theta)$$

The framework decomposes this inference into three components:

1. Physical Likelihood ($P(\xi_H | \xi_R, \theta)$):
   Modeled using a Boltzmann noisily-rational model. It assumes the human's correction $\xi_H$ represents an improvement over the robot's trajectory $\xi_R$ based on the true reward $\theta$, penalized by the effort required to make the correction.

2. Language Likelihood ($P(l | \xi_H, \xi_R, \theta)$):
   Instead of modeling the raw distribution of utterances, QuickLAP introduces a latent proxy variable $\mu_t = \theta - \theta_t$, representing the desired reward shift. A dual-Language Model (LM) pipeline processes the utterance in the context of the physical correction:

   • Attention LM ($LM_{att}$): Identifies which reward features are relevant to the user's intent, producing an attention mask $r \in \{0, 1\}^d$.
   • Preference LM ($LM_{pref}$): Determines the magnitude and direction of the shift for attended features, outputting a shift vector $\mu$ and a confidence vector $m$.
     The language likelihood is modeled as a Gaussian distribution centered on the LM-predicted shift $\mu_t$, with variance modulated by the confidence score $m$. High confidence results in low variance (high trust), while low confidence increases variance, reducing the language signal's influence.

3. Conditional Prior ($P(\theta | \hat{r})$):
   The attention mask $\hat{r}$ derived from language is used to modulate the prior over $\theta$. If a feature is not attended to (low $r$), the prior precision is high, anchoring the weight to its current value. If a feature is attended to (high $r$), the prior precision is lower, allowing the weight to adapt more freely.

Closed-Form Update Rule

By combining these components, QuickLAP derives a Maximum A Posteriori (MAP) update rule for each feature $i$:
$$\hat{\theta}_{t+1}^i = \hat{\theta}_t^i + \kappa_i(m_t^i, \hat{r}_t^i) \left[ \sigma_{L,i}^2(m_t^i) \Delta\Phi_i + \mu_t^i \right]$$
Where:

• $\Delta\Phi_i$ is the feature difference between the human correction and the robot plan.
• $\mu_t^i$ is the language-suggested shift.
• $\kappa_i$ is a gain term that adaptively weights the contributions of physical and language signals based on attention and confidence.

This formulation allows QuickLAP to robustly handle ambiguous feedback: vague language is interpreted in the context of physical corrections, and unclear physical corrections are disambiguated by language.

Key Contributions

1. Efficient Bayesian Framework: A closed-form Bayesian method for jointly interpreting physical corrections and natural language feedback in real-time, extending Inverse Reinforcement Learning (IRL) to multimodal settings.
2. LM-Based Semantic Parser: A procedure using Large Language Models (LLMs) to map free-form language to structured reward signals (attention masks, shift vectors, and confidence scores) without task-specific training.
3. Robustness to Ambiguity: The method explicitly handles the "ambiguity gap" where physical actions are opaque in intent and language is vague, by fusing them probabilistically.
4. Empirical Validation: Extensive simulations and user studies demonstrating significant improvements over baselines.

Experimental Results

Simulated Experiments

The authors evaluated QuickLAP in two domains: a robotic manipulation task (Robosuite) and an autonomous driving simulator (InterACT).

• Robot Manipulation: In a task involving obstacle avoidance and goal progress, QuickLAP reduced the Normalized Mean Squared Error (NMSE) of learned reward weights by over 2x compared to physical-only baselines.
• Driving Scenarios: Across four increasingly complex environments (varying numbers of lanes, cones, puddles, and cars), QuickLAP consistently outperformed physical-only and heuristic multimodal baselines. It reduced reward inference error by over 70% in moderate cases (e.g., avoiding cones vs. puddles) and converged significantly faster, stabilizing after fewer interventions than baselines.
• Ablation Studies: The "Language Only" variant (using physical context for LLM prompting but no physical update) performed well in simulation, suggesting LLMs can infer intent from context. However, the full QuickLAP model was shown to be more robust in scenarios with noisy feedback or ambiguous language.

User Studies

Two user studies were conducted to validate the approach with real humans:

1. Pilot Study (Experts): 12 expert users controlled a robotic arm. QuickLAP variants significantly outperformed the physical-only baseline in terms of NMSE.
2. Non-Expert Study (Driving): 15 non-expert users controlled a virtual car.
   • Subjective Metrics: Participants rated QuickLAP as significantly more understandable ($p=0.023$) and collaborative ($p=0.029$) than the physical-only baseline.
   • Preference: Participants significantly preferred QuickLAP over both the physical-only and language-only baselines.
   • Objective Performance: The behaviors learned by QuickLAP had significantly lower NMSE compared to baselines, indicating users could teach the robot their intended goals more effectively.
   • Edge Cases: The study highlighted QuickLAP's ability to resolve contradictory inputs (e.g., a user saying "stay away" while physically moving toward an obstacle) by relying on the confidence-weighted fusion, preventing incorrect reward shifts.

Significance and Claims

The paper claims that QuickLAP offers a general framework for preference learning in any domain where users can "show and tell." Its significance lies in:

• Real-Time Adaptation: Unlike methods requiring large offline datasets, QuickLAP supports online learning with sparse, ambiguous inputs.
• Principled Fusion: It moves beyond heuristic fusion by treating language as a probabilistic observation, providing a mathematically grounded way to resolve ambiguity between modalities.
• Improved Human-Robot Interaction: By making the robot's learning process more understandable and collaborative, QuickLAP facilitates more intuitive and personalized interactions, applicable to domains ranging from assistive robotics to collaborative drones.

The authors remain modest regarding limitations, noting that the framework assumes reasonably calibrated LLM outputs and currently focuses on pre-defined features. They suggest future work could explore additional modalities (gaze, gesture) and dynamic feature generation.