Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning

The paper introduces ContextMatters, a framework that combines Large Language Models with classical planning to dynamically relax goals based on environmental context, significantly improving 3D scene planning success rates and enabling successful real-world robot deployment.

The Gist

Imagine you are a robot chef in a busy kitchen. Your human boss gives you a very specific order: "Bring me a fresh, hot cup of coffee and a chocolate croissant from the bakery."

You rush to the bakery, but when you get there, you find two problems:

1. The coffee machine is broken (no coffee).
2. The bakery ran out of croissants (no chocolate ones).

Here is how different types of "brains" would handle this disaster:

• The "Pure Logic" Brain (Classical Planning): This brain is like a strict rule-following accountant. It checks the list, sees "Coffee" and "Croissant," realizes they are missing, and immediately stops. It says, "Error: Task Impossible. I cannot proceed." It refuses to change the plan, even if it means you go back empty-handed.
• The "Pure Imagination" Brain (LLM-only): This brain is like a daydreaming artist. It ignores the broken machine and the empty shelf. It confidently tells you, "I will go to the bakery, get the coffee and croissant, and bring them to you!" It creates a beautiful plan in its head, but when it tries to execute it in the real world, it crashes because the objects don't actually exist.
• The "Context Matters" Brain (This Paper's Solution): This is the smart, adaptable brain. It sees the broken machine and the empty shelf, but instead of giving up or lying, it thinks: "Okay, the boss wants a hot drink and a sweet treat. Since coffee is gone, maybe tea will do. Since croissants are gone, maybe a muffin is okay. Let's check if we have tea and muffins."

The Core Idea: "Context Matters"

The paper introduces a system called ContextMatters. It's a way to teach robots to be flexible problem solvers rather than rigid rule-followers or dreamers.

Think of it like a GPS navigation app that gets smarter when you hit a roadblock.

• Old GPS: "You are stuck. The road is closed. You are now lost."
• ContextMatters GPS: "The road is closed. No problem! I see a detour through the park. It's a bit longer, but it gets you to the same destination. Or, if the park is also closed, maybe we can go to a nearby coffee shop instead of the original one. Let's find the best possible version of your trip."

How It Works (The Two-Step Dance)

The system uses a "two-dimensional" approach to fix broken plans:

1. The "What" (Relaxation): If you can't get the exact item, can you get something similar?

   • Original Goal: "Bring me a red apple."
   • Relaxed Goal: "Bring me any apple." (Maybe the red ones are gone, but green ones are there).
   • Even More Relaxed: "Bring me any fruit."

2. The "Where" (Shifting): If the item isn't in the kitchen, is it in the pantry?

   • Original Goal: "Get the fork from the top drawer."
   • Shifted Goal: "Get the fork from the silverware drawer." (Maybe the top drawer is locked, but the bottom one is open).

The robot uses a Large Language Model (LLM) (the "common sense" part) to suggest these new, flexible goals. Then, it uses a Classical Planner (the "logic" part) to double-check: "Wait, do we actually have a green apple in the pantry? Yes? Great! Let's make a plan to get it."

Why This is a Big Deal

In the real world, things rarely go exactly as planned. Robots often fail because they are too rigid. If a robot is told to "pick up the blue cup" and the blue cup is broken, a normal robot might just spin in circles or crash.

ContextMatters allows the robot to say: "The blue cup is broken. I see a red cup nearby. The user probably just wants a cup. I will grab the red cup instead."

The Results

The researchers tested this on a real robot (a TIAGo robot arm) and in computer simulations.

• The Result: Their system was 52% more successful than the current best methods.
• Real World Test: They asked the robot to "Bring 4 snacks to the table." The robot found only 3 snacks. Instead of failing, it used common sense to swap the missing snack for a soda can (since both are "refreshments for a kid") and successfully completed the task.

The Takeaway

This paper teaches us that for robots to truly live and work among us, they need to stop being perfect planners and start being pragmatic problem solvers. They need to understand that context matters—sometimes the perfect solution doesn't exist, but a "good enough" solution that actually works is far better than a perfect plan that fails.

Technical Summary

Here is a detailed technical summary of the paper "Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning."

1. Problem Statement

Embodied agents (robots) face a critical challenge in real-world 3D environments: the gap between user intent and environmental constraints.

• Pure LLM Planners: Leverage commonsense reasoning but often "hallucinate" preconditions or actions, proposing sequences that are unfeasible or unsafe when executed in a specific scene.
• Pure Classical Planners (PDDL): Offer formal guarantees and safety but are brittle. If a goal is unsatisfiable due to missing objects or blocked paths (e.g., a drawer is locked, or a specific fork is missing), the planner simply fails, offering no mechanism to adapt the goal while preserving the user's original intent.
• The Gap: Existing hybrid approaches (like DELTA) often fail when the generated symbolic domain does not perfectly align with the physical scene, leading to execution failures without a principled way to "relax" the goal to a feasible alternative.

2. Methodology: ContextMatters

The authors propose ContextMatters, a framework that fuses Large Language Models (LLMs) with classical PDDL planning to perform hierarchical goal relaxation. The system operates on 3D Scene Graphs (3DSG) and utilizes a bidimensional search strategy.

Core Concepts

The framework defines two operators to navigate the space of possible plans:

1. Situational Shift ($\Gamma_{shift}$): Adapts the planning domain ($\Sigma$) to the specific scene context. It refines the representation of objects, predicates, and initial states based on the 3DSG to ensure the domain matches reality.
2. Goal Relaxation ($\Delta_{rel}$): When a goal is unachievable, this operator generates a functionally equivalent but weaker goal ($G'$) that preserves the user's intent.
   • Examples of relaxation: Dropping conjuncts (e.g., "2 forks" $\to$ "1 fork"), replacing specific constants with types (e.g., "fork_1" $\to$ "any fork"), or generalizing predicates.

The Architecture

The system operates via an iterative loop (Algorithm 1) involving:

1. Domain Generation: An LLM generates a PDDL domain based on the 3DSG and the initial goal.
2. Iterative Refinement: A symbolic planner attempts to solve the problem. If it fails, a Symbolic Validator (VAL) provides natural language feedback on errors (syntax, missing preconditions). The LLM uses this feedback to refine the PDDL problem.
3. Grounding Check: If a plan is found, it is virtually executed against the 3DSG to ensure no "hallucinated" objects (objects not present in the scene) are referenced.
4. Goal Shifting & Relaxation: If refinement and grounding fail, the system triggers Goal Shifting (finding alternative objects in the scene, e.g., swapping a blocked drawer for a shelf) and Goal Relaxation (broadening the goal, e.g., "3 snacks" $\to$ "2 snacks").
5. Search Strategy: The system traverses a Relaxation Graph ($G_{relax}$). It prioritizes horizontal moves (shifting the domain to match the scene) before vertical moves (relaxing the goal), ensuring the minimal necessary modification to achieve feasibility.

3. Key Contributions

• Contextual Goal-Relaxation Formalism: A novel mathematical framework defining goal relaxation along two axes: functionality (what to achieve) and feasibility (where/how to achieve it).
• The ContextMatters Framework: A planning system that couples LLM commonsense for goal proposal with classical planning for feasibility validation. It introduces a feedback loop where symbolic validators correct LLM hallucinations.
• New Dataset: A benchmark of 141 tasks across 10 environments designed specifically to be "relaxation-prone" (i.e., tasks where the original goal is impossible, requiring adaptation). This is significantly larger than previous benchmarks (e.g., DELTA's 15 tasks).
• Real-World Validation: Successful deployment on a TIAGo robot, demonstrating the system's ability to handle physical constraints and execute adapted plans in a real home environment.

4. Experimental Results

The authors evaluated ContextMatters against state-of-the-art baselines: LLMAsPlanner, SayPlan, and DELTA.

• Success Rate (SR):
  • ContextMatters (with Relaxation): Achieved a 91.73% Success Rate (Planning + Grounding).
  • DELTA (SOTA): Achieved only 39.28% Success Rate.
  • Improvement: The paper reports a +52.45% improvement over the best LLM+PDDL baseline.
• Ablation Studies:
  • Removing the "Relaxation" component dropped the SR to 66.94%, proving that the ability to adapt goals is critical for handling impossible tasks.
  • Removing "Domain Generation" (using a fixed domain) still yielded high performance (91.54%), showing the robustness of the relaxation mechanism.
• Efficiency: While ContextMatters takes longer to plan (avg. 19s vs. 0.03s for DELTA) due to the iterative refinement and LLM inference, it produces executable plans, whereas faster baselines often fail at the grounding or execution stage.
• Real Robot Demo: The system successfully handled a task "Bring 4 kid snacks to table_2" in a real room. When only 3 snacks were available, the system autonomously shifted the goal to include a "cola can" (a functional substitute for a snack in this context, adhering to safety constraints) and executed the plan.

5. Significance

• Bridging the Sim-to-Real Gap: The paper demonstrates that for robots to operate in unstructured real-world environments, they must move beyond rigid goal satisfaction. They need the ability to reason about what is possible given the current context.
• Robustness over Optimality: ContextMatters prioritizes finding a feasible plan that preserves intent over finding the perfect plan for an idealized world. This is a crucial step toward practical autonomy.
• Hybrid Intelligence: It effectively leverages the strengths of both paradigms: LLMs for semantic flexibility and commonsense reasoning, and classical planners for rigorous logical consistency and safety guarantees.
• Future Direction: The work suggests that future embodied AI systems must incorporate "context-aware partial achievement," where failure is treated as a cue to modify the goal rather than a terminal error.