💻 computer science

LENS: LLM-guided Environment Simplification for Planning and Control in Clutter

The paper introduces LENS, a plug-and-play framework that leverages Large Language Models to automatically generate and dynamically update task-specific scene abstractions by merging or pruning objects, thereby significantly improving the performance of various planning and control methods in highly cluttered robotic manipulation environments.

Original authors: Aileen Liao, Rachel Holladay, Dinesh Jayaraman, Michael Posa

Published 2026-07-23

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Aileen Liao, Rachel Holladay, Dinesh Jayaraman, Michael Posa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to tidy up a messy bedroom. To us, a messy room is just a pile of clothes, books, and toys. But to a robot's brain, it's a terrifying explosion of math. Every single object is a potential obstacle, a thing that could bump into the robot, or a thing the robot might accidentally knock over. The robot has to calculate how every single item interacts with every other item to figure out how to move. If there are too many things, the math becomes so huge that the robot freezes, like a computer trying to open a file that is too big. This is the "clutter problem," and it's the main reason why robots are great at playing in clean, empty labs but terrible at helping out in our real, messy homes. Scientists have been trying to teach robots to ignore the junk, but usually, that requires a human to manually program the robot to ignore specific things, which doesn't work well when the mess changes.

Enter a new idea called LENS (LLM-guided Environment Simplification). Think of LENS as a super-smart, magical pair of glasses for the robot. Instead of trying to solve the math for the whole messy room, the robot puts on these glasses, which use a powerful "brain" (a large language model) to look at the scene and ask, "What actually matters for this specific job?" If the robot needs to pick up a red cup, the glasses might tell it, "Ignore the pile of laundry in the corner, and treat that stack of three books as one single block." By simplifying the world in real-time, the robot can stop panicking and start working. This paper shows that by using this "smart glasses" approach, robots can handle much messier rooms than before, whether they are using old-school math-based planning or brand-new AI that learns from watching videos.

The Robot's Nightmare: Too Much Stuff to Think About

You know how when you have a huge pile of homework, it's hard to focus because you're staring at the whole mountain? Robots feel the same way. When a robot tries to move an object in a cluttered room, it has to think about every single thing in the room. Is that chair going to block the path? Will that toy roll away? If there are 50 objects, the robot has to run millions of calculations to make sure it doesn't crash.

For a long time, researchers tried to solve this by building robots that only work in very clean, organized rooms. But that's not real life. Real life is messy. When robots are put in messy rooms, they get confused. Their "vision" gets cluttered, and their "brain" gets overwhelmed. They might try to move a toy that isn't even part of the task, or they might freeze because the math is too hard. The problem isn't that the robot is stupid; it's that it's trying to do too much at once.

The Solution: A Magical Filter

The authors of this paper, Aileen Liao and her team from the University of Pennsylvania, came up with a clever fix called LENS. Instead of trying to make the robot smarter at handling everything, they decided to make the world smaller for the robot.

Imagine you are playing a video game, but the screen is filled with so many characters and items that you can't see the goal. You put on a filter that blurs out everything that isn't important. Suddenly, you can see the path clearly. LENS does exactly that for robots. It uses a special type of AI (called a Vision-Language Model) to look at the scene and decide what to keep and what to throw away.

LENS does two main things to simplify the scene:

Pruning (Throwing Away): If an object is far away or has nothing to do with the task, LENS tells the robot to pretend it doesn't exist. For example, if the robot needs to move a green block, LENS might say, "Ignore that blue ball in the corner."
Merging (Gluing Together): Sometimes, objects are stuck together, like a stack of books. Instead of treating each book as a separate problem, LENS glues them together in the robot's mind and treats them as one single, big object. This makes the math much easier.

How It Works: The "Try, Fail, Fix" Loop

Here is the really cool part: LENS isn't just a one-time filter. It's a loop. The robot tries to do the task with the simplified view. If it fails (maybe it tried to move a stack of books and they fell apart), the robot sends a message back to the "magic brain" saying, "Hey, that didn't work!" The brain then looks at the scene again, realizes it made a mistake (maybe it pruned something it shouldn't have, or merged things that shouldn't be merged), and updates the filter. It's like a human trying to solve a puzzle, realizing a piece is in the wrong spot, and trying again.

The researchers tested this in three different ways:

Planning: They used it with a classic planning system that uses logic to figure out steps.
Control: They used it with a system that controls the robot's muscles in real-time.
Learning: They used it with a modern AI that learns by watching videos (called a Vision-Language-Action model).

What They Found

The results were pretty impressive. In their experiments, the robots with LENS were much better at handling messy rooms than the ones without it.

Speed: When the number of objects in the room increased, the robots without LENS got slower and slower. At 7 objects, they took over 4,000 seconds (more than an hour!) to figure out what to do. The robots with LENS stayed fast, taking only about 40 to 135 seconds, no matter how many objects were there.
Success: In a messy room, the robots without LENS often failed completely. With LENS, they succeeded much more often. For example, in a test where a robot had to move a green bowl to a red plate, the robot without LENS got confused by other bowls in the way. The robot with LENS ignored the extra bowls and got the job done.
Real Robots: They even tested this on real physical robots, not just computer simulations. The robots were able to push objects through clutter and pick up plush toys, successfully ignoring the distractions.

Why This Matters

This paper suggests that we don't necessarily need to build "smarter" robots to handle the real world. Instead, we can teach them to be better at ignoring things. By giving robots a way to dynamically simplify their world based on what they are trying to do, we can make them much more useful in our messy, cluttered homes. It's a bit like how a good teacher helps a student focus on the most important parts of a lesson, rather than trying to memorize the whole textbook at once.

The authors are careful to say that this isn't a perfect solution yet. Sometimes the robot still makes mistakes, and the "magic brain" might need a few tries to get the filter right. But it's a huge step forward. It shows that by combining the ability to see and understand language with the ability to simplify a scene, we can bridge the gap between robots that work in labs and robots that can actually help us in our daily lives.

Technical Summary: LENS – LLM-guided Environment Simplification for Planning and Control in Clutter

Problem Statement

Despite advances in general-purpose robotic manipulation, real-world multi-object clutter remains a significant barrier to deployment. As scene complexity increases, the number of objects, contact points, decision branches, and interaction modes grows, causing computational costs and failure modes to scale poorly for both optimization-based planners and learned controllers.

Optimization and Planning Methods: Must reason over every object in the scene. Their computational cost scales with clutter even when most objects are task-irrelevant, leading to intractability.
Learned Controllers (e.g., Vision-Language-Action models): Trained on curated, tidy environments, they suffer from severe distribution shifts when deployed in dense, unstructured scenes. This leads to degraded affordance estimates and incorrect contact predictions.
Current Limitations: Existing approaches to scene abstraction often require extensive, task-specific manual engineering, which does not scale. There is a lack of a flexible, automated mechanism to suppress task-irrelevant objects before planning or control begins.

Methodology: The LENS Framework

The authors propose LENS (LLM-guided Environment Simplification), a plug-and-play, task-agnostic framework that automatically generates scene-specific, task-specific abstractions using Multimodal Large Language Models (MLLMs) and closed-loop feedback.

Core Mechanism

LENS operates as a front-end to existing planning and control stacks. Given a natural language task description ( $\tau$ ) and a full scene description ( $O$ ), a VLM (specifically GPT-4o in the implementation) is prompted to output a structured, abstracted scene representation ( $\tilde{O}$ ). This is achieved through two elementary operations:

Pruning: Objects deemed irrelevant to the task (e.g., distant distractors) are removed from the scene representation. Depending on the downstream system, this reduces state space variables, contact constraints, or visual inputs.
Merging: Objects that are functionally or dynamically coupled (e.g., a stack of objects) are grouped into a single composite entity. This reduces the number of bodies and contact pairs while retaining a conservative physical envelope of the group.

Closed-Loop Refinement

To address potential VLM hallucinations or dynamic changes in the environment, LENS employs a closed-loop re-prompting mechanism. If the downstream system fails (triggered by time-outs, error codes, or infeasible plans), the failure feedback is appended to the prompt. The VLM is then queried again to revise the scene abstraction (e.g., expanding the set of relevant objects or adjusting merges) until the task succeeds or a maximum number of retries is reached.

Integration with Downstream Systems

LENS was integrated into three distinct robotic paradigms:

Task and Motion Planning (TAMP): Pruned objects are fixed in pose and excluded from decision variables; merged groups are treated as single rigid bodies with a unified ID.
Contact-Implicit Model Predictive Control (C3+): Pruning reduces the dimensionality of the Linear Complementarity Problem (LCP) and contact pairs. Merged groups are updated in real-time based on pose, though internal shape changes are not modeled.
Vision-Language-Action (VLA) Models ( $\pi_{0.5}$ ): Pruning is implemented by inpainting excluded objects from the visual observation, creating a filtered image passed to the policy. Merging is not utilized in this specific implementation.

Key Contributions

The paper claims three primary contributions:

A General, Closed-Loop Abstraction Method: A task-agnostic framework for estimating task relevance and constructing scene abstractions using vision-language models, eliminating the need for hand-crafted heuristics.
Versatile Integration: Successful integration of this abstraction layer with three state-of-the-art paradigms: TAMP, optimization-based contact-implicit control, and large-scale VLA models.
Experimental Validation: Demonstration of improved scalability, robustness, and efficiency in highly cluttered tabletop settings across all three integrated systems.

Experimental Results

The authors evaluated LENS against baselines operating on unfiltered observations across simulation and real-robot hardware.

TAMP Performance: In heavily cluttered environments, LENS-TAMP significantly outperformed the baseline. The baseline frequently timed out due to the combinatorial explosion of distractors, whereas LENS maintained high success rates by pruning irrelevant objects and merging stacks.
Model-Based Control (C3+):
- Scalability: As object count increased, the baseline execution time degraded drastically (exceeding 4000 seconds with 7 objects). In contrast, LENS-C3+ maintained stable execution times (~40–135 seconds) regardless of clutter level.
- Success Rates: LENS achieved a 39/39 success rate in simulation trials compared to 17/30 for the baseline.
- Hardware: On real robots, iterative feedback allowed recovery from initial pruning errors, achieving an 80% cumulative success rate across three feedback loops.
VLA Performance ( $\pi_{0.5}$ ):
- Simulation: Clutter reduced VLA success from 0.85 to 0.5. LENS recovered performance to 0.69.
- Hardware: In experiments involving picking fruit plushies and placing them in bowls, LENS consistently improved success rates over the baseline across all fruit types (e.g., Pineapple: 0.5 vs 0.2; Apple: 0.7 vs 0.0).

Significance and Claims

The paper argues that the gap between laboratory algorithms and real-world deployment is not merely a matter of scale but of fundamental tractability. Without a mechanism to focus on task-relevant elements, downstream methods inherit the full combinatorial and distributional burdens of clutter.

The significance of LENS lies in its ability to:

Shift the burden of scene abstraction from manual engineering to an automated, semantic reasoning loop.
Bridge the gap between structured laboratory settings and unbounded real-world complexity.
Provide a "plug-and-play" solution that enhances existing planners and controllers without requiring retraining of the core policies.

The authors note limitations, including the computational cost of validating abstract representations in simulation for some controllers and the reliance on high-level failure signals rather than explicit causal analysis. They suggest future work could extend these ideas to richer abstraction operations (e.g., via RL) and long-horizon tasks where relevance evolves over time.