🤖 AI

STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition

STELLA is an efficient on-device Human Activity Recognition framework that achieves state-of-the-art performance and privacy-preserving personalization by compressing raw sensor data into compact latent tokens for a frozen LLM, thereby shifting the adaptation burden from the model to a lightweight, user-specific tokenizer.

Original authors: Nirhoshan Sivaroopan, Albert Zomaya, Kanchana Thilakarathna

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nirhoshan Sivaroopan, Albert Zomaya, Kanchana Thilakarathna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your smartwatch or fitness tracker is like a very observant but slightly overwhelmed assistant. It's constantly watching your body move, collecting thousands of tiny data points every second (like how fast you're walking, which way you're turning, or how hard you're jumping).

For a long time, we've tried to teach these devices to recognize what you're doing (running, sleeping, typing) by building special, narrow-minded "classifiers" for each specific task. But recently, we started using Large Language Models (LLMs)—the same kind of AI that writes stories and answers questions—because they are great at understanding context and meaning.

However, there's a big problem: LLMs are bad at raw numbers.

If you try to feed a language model a raw stream of sensor data, it's like trying to explain a complex dance to a poet by shouting out a list of 1,000 random numbers. The poet (the AI) gets confused, the message takes too long to send, and it might even forget the beginning by the time it reaches the end. Also, sending all that data to the cloud (the internet) to be processed is slow and risky for your privacy.

Enter STELLA.

STELLA is a new system that acts like a super-efficient translator right on your device. Instead of shouting a list of numbers to the AI, STELLA listens to the sensor data and summarizes it into a tiny, perfect "note" before passing it along.

Here is how it works, using some simple analogies:

1. The "Summarizer" (The Tokenizer)

Imagine you have a 10-minute video of your day. If you wanted to tell a friend what happened, you wouldn't read them the entire script word-for-word. You'd say, "I went for a run, then grabbed coffee, then worked."

STELLA does exactly this. It takes a massive window of raw sensor data (hundreds of numbers) and compresses it into just 16 tiny "tokens" (think of them as 16 very smart, compressed words).

The Magic: It doesn't just shrink the data; it understands the structure of your movement. It knows the difference between the quick "thump" of a footstep and the steady rhythm of a walk.
The Result: The AI receives a short, easy-to-read sentence instead of a wall of numbers. This makes the process incredibly fast and keeps the AI's memory usage low.

2. The "Frozen Brain" (The LLM)

Usually, when we want an AI to do a new job, we have to retrain it or shrink it down, which often makes it forget its general smarts. STELLA keeps the AI's brain frozen (unchanged).

Think of the AI as a brilliant, general-purpose librarian who knows everything about human behavior.
STELLA doesn't try to turn the librarian into a "Running Expert." Instead, it just gives the librarian a perfectly written note: "The sensor data looks like this: [16 tokens]. Based on your knowledge, is this 'Running' or 'Walking'?"
Because the note is so short and clear, the librarian can answer instantly without needing a new library card for every single activity.

3. The "Personal Assistant" (On-Device Personalization)

Everyone moves differently. You might have a unique way of jogging compared to your friend. A generic AI might get confused by your style.

STELLA has a special feature where it can learn just about you without changing the main librarian.
It tweaks only the "Summarizer" (the translator) to understand your specific gait.
It also keeps a small, private notebook on your phone with examples of your past movements. When you do something new, it quickly checks this notebook to say, "Ah, this looks like how you usually run."
Privacy Bonus: All of this happens right on your phone. Your movement data never leaves your device to go to a server.

Why is this a big deal?

The paper tested STELLA on seven different datasets (covering everything from walking to industrial assembly line work) and found:

It's the best: It beat all previous methods, sometimes by a huge margin (up to 11.8% better accuracy).
It's fast: It runs in real-time on regular smartphones, not just supercomputers.
It's private: Because the heavy lifting is done by the translator on your phone, your data stays with you.
It's flexible: It works whether you have a simple watch with one sensor or a complex suit with dozens of sensors.

In short: STELLA solves the problem of "How do we make a smart language model understand our body movements without slowing down or leaking our privacy?" by building a tiny, smart translator that turns messy sensor data into a clean, short story the AI can understand instantly.

Technical Summary: STELLA – Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition

1. Problem Statement

Human Activity Recognition (HAR) from Inertial Measurement Unit (IMU) sensors is critical for mobile and ubiquitous computing. While Large Language Models (LLMs) offer promising capabilities for semantic reasoning and generalization, deploying LLM-based HAR on edge devices faces three primary obstacles:

Input Mismatch and Cost: LLMs are designed for short sequences of discrete language tokens. Raw sensor data consists of long, multi-channel streams of continuous floating-point values. Serializing these directly creates prompts that are excessively long, inflating the LLM's quadratic attention costs and KV-cache memory usage beyond edge device budgets.
Loss of Generality: Fine-tuning or rewriting an LLM to ingest raw sensor data directly transforms a general-purpose model into a task-specific classifier. This defeats the purpose of edge deployment, where devices typically host a single, shared LLM for multiple modalities (text, voice, vision) rather than maintaining multiple specialized models.
Privacy and Latency: Offloading sensor data to cloud-based LLMs introduces network dependency, unbounded latency, and significant privacy risks by transmitting sensitive behavioral data off-device.

Existing approaches either rely on cloud APIs (sacrificing privacy and real-time guarantees) or attempt to compress/fine-tune the LLM itself (sacrificing reusability and generalizability). There is a lack of an efficient interface that allows a frozen, general-purpose LLM to reason over sensor data without modification.

2. Methodology: STELLA

STELLA (Sensor Tokenization for Edge LLM Activity Recognition) reframes the problem from "LLM adaptation" to "sensor tokenization." The core architecture consists of three main components:

2.1 Hierarchical Sensor Tokenizer (STELLA-Tower)

Instead of processing raw numerical streams, STELLA employs a lightweight, hierarchical encoder that compresses an entire multi-channel sensor window ( $C \times T$ ) into a fixed set of $N$ latent sensor tokens (default $N=16$ ).

Convolutional Stack: Initial depthwise and pointwise convolutions extract local temporal patterns and mix cross-channel information while reducing temporal resolution.
Hierarchical Stages: The encoder uses three stages. Early stages utilize RepMixer blocks (reparameterizable mixers based on convolutions) for efficient local feature extraction. A final Multi-Head Self-Attention (MHSA) layer captures long-range dependencies on the already downsampled sequence.
Multi-Scale Aggregation: Features from three hierarchical levels (local motion patterns, activity dynamics, and global context) are pooled to a fixed token count via linear interpolation and concatenated.
Key Property: The output is a fixed-length sequence of tokens regardless of the number of sensor channels or the window length, ensuring the downstream LLM cost remains invariant to sensor configuration.

2.2 Projection and Integration

The latent sensor tokens are projected into the embedding space of a frozen, pretrained LLM (e.g., GPT-2) using a shallow Multi-Layer Perceptron (MLP). These projected tokens are concatenated with a natural-language task prompt (e.g., "Predict the human activity from the sensor data"). The LLM remains frozen; only the tokenizer and projection layers are trained.

2.3 Inference via Label Scoring

Rather than generating free-form text, STELLA treats classification as a label-scoring problem. For a closed set of candidate activity labels, the system calculates the average per-token log-likelihood (Negative Log-Likelihood) assigned by the frozen LLM to each candidate's token sequence. The label with the highest score is selected. This approach avoids the unpredictability of generative outputs and keeps the prompt length short.

2.4 On-Device Personalization

STELLA supports personalization without fine-tuning the LLM. After deployment, the lightweight sensor tokenizer is adapted to a specific user's data using a small amount of labeled samples. Additionally, a local retrieval mechanism augments inference:

A knowledge base of the user's past sensor-token embeddings and labels is maintained on-device.
During inference, the top- $k$ most similar historical contexts are retrieved and prepended to the current input.
This allows the system to adapt to individual gait and motion styles while keeping all user data and context strictly on-device.

3. Key Contributions

Reframing the Bottleneck: The paper argues that the efficiency bottleneck for LLM-based HAR is the sensor-to-LLM interface, not the LLM itself. It proposes solving this via a learned sensor tokenizer rather than compressing the LLM.
Configuration-Invariant Tokenization: The STELLA-Tower compresses variable-length, multi-channel sensor windows into a fixed set of latent tokens ( $N=16$ ). This decouples LLM inference cost from sensor configuration, reducing token counts by up to $262.5\times$ compared to scalar-level encoding.
Private, Lightweight Personalization: The framework enables on-device adaptation by updating only the tokenizer (approx. 10% of parameters) and utilizing a local retrieval context, preserving the LLM as a reusable, frozen backend.
State-of-the-Art Performance with a Small LLM: STELLA achieves new SOTA results across seven public HAR datasets using only a frozen GPT-2 backbone, outperforming methods that rely on much larger LLMs or cloud-based APIs.

4. Experimental Results

The authors evaluated STELLA on seven datasets (HHAR, PAMAP2, MHEALTH, GOTOV, SKODA, Opportunity, USC-HAD) covering 6 to 60 channels and 6 to 18 activity classes.

Accuracy: STELLA achieved new SOTA performance on all eight benchmark settings. Improvements over prior methods ranged from marginal on saturated benchmarks to substantial on sparse-sensor datasets (e.g., +11.83% F1 on HHAR, +9.20% F1 on USC-HAD).
Tokenizer Efficacy: When compared against representative time-series encoders (PatchTST, Chronos, TimesNet, etc.) under the same LLM pipeline and token budget, STELLA's tokenizer consistently outperformed them, confirming that the gains stem from the specific sensor-to-LLM translation design.
On-Device Feasibility:
- Latency: Inference is real-time on commodity hardware. On a server GPU, latency is ~9.7 ms; on a flagship smartphone (Samsung S23 Ultra) with GPU acceleration, it is ~41.7 ms. Even on a single-thread CPU, it meets real-time constraints for standard stride budgets.
- Memory: The full model (FP32) requires ~555 MB, fitting within the memory constraints of modern mobile GPUs (1 GB+ VRAM).
Personalization: Adapting the tokenizer and using retrieval augmentation improved the F1 score for the most difficult subjects by up to 21.91% (e.g., from 70.79% to 92.70% on USC-HAD) with less than 1 ms of retrieval overhead.

5. Significance and Claims

The paper claims that STELLA provides a practical path toward accurate, private, and personalized LLM-based HAR on edge devices. Its significance lies in:

Preserving the General-Purpose LLM: By shifting the adaptation burden to the sensor side, STELLA allows HAR to be treated as just another input modality for a shared, system-level local LLM, rather than requiring a separate, fine-tuned model stack.
Privacy and Efficiency: It enables continuous, real-time HAR entirely on-device, eliminating the need to stream sensitive behavioral data to the cloud.
Scalability: The fixed-token interface ensures that inference latency and memory usage remain predictable regardless of the number of sensors or the length of the data window, making the system robust to varying hardware configurations.

The authors conclude that efficient sensor tokenization, rather than LLM compression or cloud offloading, is the key to unlocking the potential of LLMs for edge-based sensing.