💻 computer science

Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals

This paper introduces SCOTTI, a novel shared convolutional transformer framework that leverages multi-task learning on a new tactile dataset to simultaneously improve 3D human pose estimation, action classification, and movement progress prediction from foot insole signals, outperforming existing single-task approaches while addressing vision-based limitations like occlusion and privacy.

Original authors: Isaac Han, Seoyoung Lee, Sangyeon Park, Ecehan Akan, Yiyue Luo, Joseph DelPreto, Kyung-Joong Kim

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Isaac Han, Seoyoung Lee, Sangyeon Park, Ecehan Akan, Yiyue Luo, Joseph DelPreto, Kyung-Joong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human movement. Usually, we teach robots by showing them videos through cameras. But cameras have two big problems: they can't see through walls or if someone blocks the view (occlusion), and they make people feel like they are being watched (privacy).

This paper introduces a clever new way to teach robots: by listening to the "footprints" of pressure.

Here is the breakdown of their invention, SCOTTI, using simple analogies.

1. The "Smart Insole" (The Ears of the Robot)

Instead of a camera, the researchers gave people special, low-cost insoles (like shoe inserts) packed with over 500 tiny pressure sensors.

The Analogy: Think of these insoles as a super-sensitive "touch microphone." Just as a microphone picks up sound waves, these insoles pick up the "pressure waves" of your feet.
The Magic: They are wireless and cheap (about $50), so anyone can wear them without needing a fancy studio or a camera crew.

2. The Three Tasks (The Three Questions)

The researchers wanted the robot to answer three different questions at the same time based on these pressure signals:

3D Pose: "Where exactly are your body parts?" (Is your knee bent? Is your arm up?)
Action Classification: "What are you doing?" (Are you squatting, jumping, or walking?)
Progress Prediction: "How far along are you?" (Are you halfway through a squat, or just starting?)

The Problem: Before this paper, scientists tried to teach the robot these three things separately. It was like hiring three different teachers: one for math, one for history, and one for art. They didn't talk to each other, so the student (the robot) learned slowly and made mistakes.

3. The Solution: SCOTTI (The "Super-Student")

The team built a new AI model called SCOTTI (Shared COnvolutional Transformer for Tactile Inference).

The Analogy: Imagine a single, super-smart student who takes all three classes at once. Because the student learns that "squatting" (Action) involves "bending knees" (Pose) and "going down then up" (Progress), learning one helps them get better at the others.
How it works: The model uses a mix of two powerful AI tools:
- CNNs (The Detective): Looks at the shape of the pressure on the foot (spatial).
- Transformers (The Storyteller): Understands the sequence of events over time (temporal).
- Shared Brain: Instead of three separate brains, SCOTTI has one shared "brain" that learns a common language of movement, then branches out to answer the three specific questions.

4. The "Gym Class" Dataset

To train this AI, they didn't just use a few people. They gathered 15 volunteers and filmed them doing 8 different activities (squats, lunges, jumping, walking, etc.) for 7 hours.

The Catch: They used cameras only to write down the "correct answers" (the ground truth) while the people wore the insoles. Once the robot is trained, it doesn't need the cameras anymore—it only needs the shoe sensors.
The Result: They created a massive library of 200,000+ synchronized "pressure + video" moments, which they are releasing to the public so others can learn from it.

5. The Results: Why It Matters

When they tested SCOTTI, it beat all the old methods that tried to do the tasks separately.

Better Accuracy: It guessed body positions more precisely.
New Capability: It was the first to successfully predict how far along an action was using just foot pressure (a task no one had done before).
Efficiency: It did all three jobs with one model, saving computing power and time.

The "Aha!" Moment: What Did the AI Learn?

The researchers peeked inside the AI's "brain" to see what it was thinking.

The Center is Key: They found the AI cared most about the center of the foot (the arch and ball), not the heels or toes. This makes sense because the center of the foot is where you balance and push off.
The "Ring" of Walking: When they visualized the data for walking, it looked like a circle. As you walk, your left foot lifts, then your right foot lifts, creating a perfect loop. The AI learned this pattern naturally.

Summary

SCOTTI is like a robot that can "feel" your movement through your shoes. By teaching it to solve three puzzles at once (Where am I? What am I doing? How far have I gone?), it became much smarter than robots taught one puzzle at a time. This technology could help robots assist the elderly, analyze sports performance, or interact with humans in private spaces without ever needing a camera.

1. Problem Statement

While vision-based methods for human pose estimation, action classification, and movement progress prediction are advanced, they suffer from significant limitations in real-world environments:

Occlusion: Cameras cannot see through obstacles or when body parts are hidden.
Privacy: Continuous video monitoring raises privacy concerns.
Infrastructure: Vision systems often require dedicated, instrumented spaces.

Tactile sensing offers a robust alternative by capturing force and pressure data directly related to posture without privacy or occlusion issues. However, existing tactile-based research typically handles pose estimation, action classification, and progress prediction as separate, isolated tasks. This independent approach fails to leverage the inherent correlations between these tasks, leading to suboptimal performance. Furthermore, action progress prediction (estimating how far along an action is) has not been previously explored using foot tactile signals.

2. Methodology: SCOTTI

The authors propose SCOTTI (Shared COnvolutional Transformer for Tactile Inference), a unified multi-task learning framework.

A. Network Architecture

SCOTTI combines Convolutional Neural Networks (CNNs) for spatial feature extraction and Transformers for temporal modeling:

Input: A sequence of $T$ tactile frames (sliding window) from left and right insoles. Each frame is a 2D grid ( $H \times W$ ) representing foot pressure.
Spatial Encoding (CNN): Each frame is processed independently by a CNN feature extractor (two convolutional layers with ReLU and pooling) to capture spatial pressure distributions. The output is flattened and projected into an embedding space ( $E$ ).
Temporal Encoding (Transformer):
- The $T$ frame embeddings are concatenated.
- A learnable Class Token is appended to the sequence to aggregate global information.
- Learnable Positional Encodings are added to encode temporal order.
- A Transformer Encoder processes the sequence to capture long-range temporal dependencies.
Shared Representation: A weighted mean pooling is applied to the Transformer output to generate a single shared feature vector representing the tactile signal.
Task Heads: This shared representation is fed into three separate Multi-Layer Perceptrons (MLPs) to predict:
- 3D Human Pose: 19 keypoints (head to toes).
- Action Class: Categorical classification (8 classes).
- Action Progress: Continuous regression value $p \in [0, 1]$ .

B. Loss Function

The model is trained using a weighted sum of three loss functions:
$L_{total} = w_{pose} L_{pose} + w_{action} L_{action} + w_{progress} L_{progress}$

Pose Loss: Sum of Mean Per Joint Position Error (MPJPE) and Mean Per Joint Angle Error (MPJAE).
Action Loss: Standard Cross-Entropy.
Progress Loss: Mean Squared Error (MSE).

3. Key Contributions

A. Novel Unified Model (SCOTTI)

SCOTTI is the first model to simultaneously learn 3D pose, action classification, and action progress from tactile signals. It demonstrates that multi-task learning with shared representations significantly outperforms single-task baselines.

B. First Action Progress Prediction via Tactile Signals

The paper introduces the concept of predicting continuous action progress (0% to 100%) using foot pressure. The authors define a mathematical framework to label progress based on periodic motion indicators (e.g., hip height for squats, heel distance for walking).

C. Large-Scale Dataset

The authors collected a novel, open-source dataset comprising:

Participants: 15 individuals.
Duration: Over 7 hours of synchronized data.
Volume: >200,000 frames.
Activities: 8 distinct actions (squat, lunge, jump, step-up, side-walking, in-place walking, backward walking, normal walking).
Hardware: A custom, low-cost (~$50), wireless insole sensor with >500 piezoresistive sensors per foot, transmitting at 16Hz. Ground truth poses were captured via a 6-camera XRmocap system.

4. Experimental Results

The model was evaluated using a leave-one-subject-out cross-validation strategy (10 train, 5 test) across three metrics: MPJPE (pose), Accuracy (action), and APP/MSE (progress).

Performance Comparison

SCOTTI outperformed all state-of-the-art baselines (including Voxel-CNN, Res-CNN, GCN-Transformer, and STAT) and their multi-task variants:

Pose Estimation: Achieved 96.63 mm MPJPE (best), significantly lower than the next best (98.51 mm).
Action Classification: Achieved 90.06% accuracy, outperforming the specialized STAT baseline (84.56%).
Progress Prediction: Achieved 0.0223 MSE and 0.8952 APP, vastly superior to random baselines and multi-task versions of other models.

Ablation Studies

Multi-task vs. Single-task: Removing the shared representation (training single-task models) resulted in significant performance drops (e.g., Pose MPJPE increased from 96.63 to 101.61 mm). This confirms the mutual benefit of the tasks.
Sensing Regions: Analysis revealed that the central foot regions are the most critical for all tasks, while edges and heels are less informative.
Feature Visualization (t-SNE): The shared feature space showed structured patterns:
- Progress: Actions with alternating foot movements (walking, lunges) formed ring-like structures where progress cycles from 0 to 1.
- Action: Features for the same action class clustered tightly, indicating the model learned discriminative representations.

5. Significance and Impact

Robust Human-Robot Interaction: By overcoming occlusion and privacy issues, SCOTTI enables robots to understand human intent and movement progress in unstructured, real-world environments (e.g., homes, hospitals).
Efficiency: A single unified model replaces the need for multiple specialized systems, reducing computational overhead and deployment complexity.
New Research Direction: The introduction of action progress prediction via tactile signals opens new avenues for applications in sports training (form correction), elderly care (fall detection and activity monitoring), and rehabilitation.
Open Science: The release of the dataset and sensor design (low-cost, wireless) lowers the barrier for future research in tactile-based human motion analysis.