🤖 machine learning

More with LESS -- Local Scene Representations for Tactile Imaging

This paper introduces LESS (Local Encoder for Spatial Sensing), an object-centric tactile representation that utilizes a grid of recurrent encoders with local receptive fields to achieve robust generalization, spatial uncertainty estimation, and full 3D reconstruction of internal soft object structures through both robot-controlled and human-like hand-held palpation.

Original authors: Zohar Rimon, Elisei Shafer, Tal Tepper, Daniel Kozin, Alon Malka, Roy Holland, Aviv Tamar

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zohar Rimon, Elisei Shafer, Tal Tepper, Daniel Kozin, Alon Malka, Roy Holland, Aviv Tamar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out what's inside a sealed, squishy water balloon without popping it. You can't see inside, but you can poke it with your fingers. If you press on one side and feel a hard lump, you know there's something solid inside. If you press elsewhere and it feels soft, that area is just water.

This is the challenge of Tactile Imaging: using touch to "see" the hidden insides of soft objects, like tumors inside a breast or defects inside a soft robot.

Here is a simple breakdown of what the researchers at the Technion (Israel Institute of Technology) did to solve this problem.

The Old Way: The "One Big Brain" Problem

Previous attempts to do this used a robot arm to poke the object and a computer to guess the shape. However, the computer used a "global" approach. Think of this like trying to describe a whole city using a single, giant sentence. If you want to describe a city with two parks, the computer has to learn every possible combination of "two parks" from scratch. If you show it a city with three parks, or a city that is twice as big, the computer gets confused because it never saw that exact combination before. It's rigid and struggles to generalize.

The New Way: LESS (Local Encoder for Spatial Sensing)

The researchers proposed a new method called LESS. Instead of one giant brain trying to remember the whole object at once, they gave the computer a team of tiny, local detectives.

The Analogy: Imagine a large map of a city. Instead of one person trying to memorize the whole map, you place a small team of detectives at every street corner.
How it works: Each detective only looks at what is happening in their immediate neighborhood (their "receptive field"). They don't care what's happening three blocks away.
The Magic: If you have a city with one park, the detective near the park learns what a park feels like. If you later show them a city with three parks, they don't panic. They just say, "Hey, I see a park here, and my neighbor sees a park over there." Because they work independently, they can combine their findings to describe complex shapes they've never seen before. This is called compositional generalization.

What They Actually Built and Tested

The paper details several specific achievements:

A Massive New Dataset: They built a robot setup that automatically poked hundreds of soft, silicone "phantoms" (fake models of body parts) while taking MRI scans to know the "true" answer. This is the largest dataset of its kind.
From 2D to 3D: Previous methods could only guess a flat, 2D slice of the inside. LESS can reconstruct the full 3D volume, giving a much clearer picture of the object's shape and size.
The "Hand-Held" Breakthrough: The old robot methods required the sensor to be held by a precise robot arm. The researchers figured out how to make this work with a hand-held device (like a doctor holding a probe).
- They added a special tracker to follow the hand's movement.
- They trained the AI on data that mimics human hand motions (which are wobbly and varied) rather than just perfect robot movements.
Uncertainty Maps: The system doesn't just guess; it tells you how confident it is. If the detective hasn't been poked enough in a specific area, the system highlights that spot as "uncertain," telling the operator, "Go poke here again to be sure."

The Results

Better Accuracy: When tested on objects with multiple lumps (inclusions) or larger sizes that the AI had never seen during training, LESS worked significantly better than the old "global" method. The old method often failed completely on these new shapes, while LESS successfully identified them.
Real-Time Imaging: They demonstrated a working prototype where a user holds the sensor, moves it over an object, and sees the internal structure appear on a screen in real-time, complete with a "confidence map" showing which areas need more poking.

What They Did Not Claim

It is important to stick to what the paper says:

They tested this on synthetic silicone models (phantoms), not on actual human patients.
While they mention medical applications as a goal, they did not test this on humans or claim it is ready for clinical diagnosis yet.
They did not claim it works on any object in the world, but specifically on soft, deformable objects with internal structures similar to their training data.

In short, the researchers built a smarter, more flexible way for computers to "feel" inside soft objects, moving from rigid robot arms to a flexible, hand-held system that can handle complex shapes it has never seen before.

Technical Summary: More with LESS – Local Scene Representations for Tactile Imaging

Problem Statement
Tactile imaging (TI) aims to reconstruct the internal structure of soft objects through touch, a capability critical for medical diagnosis (e.g., detecting lesions) and robotic manipulation. While recent self-supervised learning (SSL) approaches, such as the work by Rimon et al. [34], have demonstrated promising results, they suffer from significant limitations:

Global Representations: Existing methods encode entire tactile scenes into a single, unstructured vector. This ignores the compositional nature of physical objects, requiring training on "all possible object combinations" to generalize, which is computationally expensive and often infeasible.
Robot-Dependence: Prior methods rely on precise robot kinematics for pose tracking and vertical motion patterns, limiting their applicability to hand-held devices where human palpation involves complex, non-vertical movements.
Dimensionality: Previous work was largely restricted to 2D imaging, missing the richer structural information available in 3D volumes.

Methodology: LESS (Local Encoder for Spatial Sensing)
The authors propose LESS, an object-centric neural architecture designed to exploit the inherently local nature of tactile sensing. The core insight is that the tactile response of an object's internal structure at a specific position is nearly independent of structures at distant positions.

Architecture:
- Grid of Particles: Instead of a single global vector, the tactile scene is modeled as a grid of $N$ local representations (particles). Each particle is a recurrent neural network (specifically a Gated Recurrent Unit, GRU) with a defined spatial receptive field.
- Local Processing: Each particle processes only the sequence of tactile measurements (force and pose) falling within its receptive field. Inputs are centered relative to the particle's position and encoded using sinusoidal positional encodings and MLPs.
- Shared Weights: All particles share weights, allowing the model to learn local tactile responses that are invariant to absolute position.
- Image Reconstruction: The latent states of all particles are fused to reconstruct 2D or 3D images. For image generation, each particle's latent state is mapped to a local image patch via a transposed convolution. These patches are stitched together by averaging logits and applying a softmax to produce the final image.
- Uncertainty Estimation: The local nature of the architecture allows for pixel-level uncertainty estimation using Shannon entropy of class probabilities, as uncertainty at one location is not entangled with distant locations.
Training and Data Collection:
- Self-Supervised Learning (SSL): The encoder-decoder is trained via force prediction. The model predicts the force at a future position given the current local representation, minimizing Mean Squared Error (MSE).
- Datasets: The authors collected the largest soft-body tactile interaction dataset to date (~800 hours), comprising:
  - data-v1: Vertical "poke" motions (similar to prior work).
  - data-poke: Vertical motions with varied sensor orientations.
  - data-primitive: Teleoperated motions resembling human palpation.
  - data-handheld: Manual data collected with a hand-held sensor tracked by a fiducial-based pose estimation system.
- Phantoms: Modular phantoms with soft silicone skins and hydrogel fillings containing inclusions of varying shapes, sizes, and numbers (single, multiple, connected). Ground truth was obtained via 3T MRI.
- Loss Functions: For image reconstruction, the authors use a combination of Focal loss and Dice loss to address class imbalance between the small inclusions and the large background gel.

Key Contributions

Zero-Shot Compositional Generalization: LESS demonstrates the ability to generalize to unseen configurations. Models trained exclusively on phantoms with single inclusions accurately reconstruct objects with multiple inclusions and larger dimensions at test time. This is achieved because the model learns local patterns rather than global scene combinations.
3D Reconstruction: The method extends tactile imaging from 2D slices to full 3D volumes. Surprisingly, training on 3D labels improves 2D slice metrics compared to 2D-only training, likely due to the model learning spatial regularities across adjacent slices.
Hand-Held Tactile Imaging: The authors present a proof-of-concept hand-held system. By incorporating external pose tracking and training on diverse motion primitives (including human-like motions), they enable real-time imaging outside of a fixed robotic setup.
Spatial Uncertainty: The architecture provides localized uncertainty estimates, identifying areas where the operator should collect additional data, a feature difficult to achieve with global representations.

Experimental Results

Generalization: In zero-shot tests on "multiple inclusion" and "large" phantoms (out-of-distribution), LESS significantly outperformed the global baseline (GLOBAL). For example, on the multiple inclusion set, LESS achieved an F1 score of 71.1% compared to 44.6% for GLOBAL.
3D vs. 2D: Training on 3D volumes yielded better 2D reconstruction accuracy (e.g., lower Center-of-Mass error) than training on 2D slices alone.
Pose Robustness: The system remains stable with position noise up to ~1 mm and orientation noise up to ~0.1 radian.
Hand-Held Performance: While robot-held sensors yielded slightly better accuracy (CoM error ~~1.6 mm) than hand-held operation (~~8.4 mm), training on diverse motion data (data-primitive) substantially reduced the performance gap compared to training only on robotic poke data.

Significance and Claims
The paper claims that LESS represents a significant step toward practical, operator-independent tactile imaging. By shifting from global to local representations, the method overcomes the data inefficiency of previous SSL approaches, enabling generalization to complex, composite objects without retraining. The development of a hand-held system with real-time uncertainty visualization bridges the gap between robotic research and clinical utility, potentially aiding in home-use devices and guiding physicians during examinations. The authors note that while the current work uses synthetic phantoms, the technical framework (particularly the hand-held system) is designed to facilitate future evaluation on human tissue.