🤖 machine learning

RAG-HAR+: Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deployment

The paper introduces RAG-HAR+, a cost-optimized, retrieval-first framework for Human Activity Recognition that leverages an offline agent to design dataset-specific features and employs majority voting to minimize LLM usage while maintaining competitive performance across diverse benchmarks.

Original authors: Hansi Karunarathna, Nirhoshan Sivaroopan, Chamara Madarasingha, Anura Jayasumana, Kanchana Thilakarathna

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hansi Karunarathna, Nirhoshan Sivaroopan, Chamara Madarasingha, Anura Jayasumana, Kanchana Thilakarathna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your body is a busy orchestra, and tiny sensors on your wrist or clothes are the conductors, constantly listening to the rhythm of your steps, the sway of your arms, and the beat of your heart. This field, called Human Activity Recognition (HAR), is how computers learn to tell the difference between you jogging, sleeping, or typing. For a long time, the best way to teach computers this was to force them to memorize huge libraries of labeled examples, like a student cramming for a test by reading every single book in the library. But this is expensive, slow, and breaks if you change the sensor or the activity. Recently, a new idea emerged: instead of memorizing, why not let the computer "look up" similar past examples in a giant database and ask a super-smart AI (a Large Language Model, or LLM) to help figure out what's happening? It's like having a genius librarian who can instantly find the most similar stories to help you solve a mystery. But even this new method has a snag: asking the genius librarian for help on every single step you take is slow, costs a lot of money, and requires a constant internet connection.

This paper introduces a clever upgrade called RAG-HAR+ that solves this problem by changing when and how we ask for help. The authors found that the previous method was like calling a consultant for every single question, even the easy ones. RAG-HAR+ acts more like a smart detective agency. First, it uses the AI offline (before you even start moving) to design a better "filing system" for the database, choosing the specific clues that matter most for your particular activity. Then, when you are moving, the system tries to solve the mystery just by looking at the most similar past examples in the database. It only calls the expensive AI consultant if the clues are confusing and the database can't agree on an answer. By doing this, the system becomes incredibly fast, cheap, and works even when the internet is spotty, while still being just as accurate as the old, slower methods.

The Problem: The "Over-Calling" Detective

Imagine you have a super-smart AI assistant who knows everything about human movement. In the original system (RAG-HAR), every time you take a step, the system stops, sends a message to the AI, and asks, "What am I doing?" The AI reads the message, looks at some past examples, and replies. This works, but it's like hiring a world-famous detective to solve every single case, from "Who stole the cookie?" to "Who murdered the butler?" It's a waste of the detective's time and money for the cookie theft. Plus, if the detective is in a different country (the cloud), you have to wait for a message to travel back and forth every time, making the whole process slow.

The researchers realized that for most steps, the answer is obvious if you just look at the right past examples. You don't need a genius to tell you that walking looks like walking; you just need to find a few other examples of walking in the database. The problem with the old system was that it used a "one-size-fits-all" way to describe the movement, which sometimes missed the subtle clues that make an activity unique.

The Solution: The Smart Filing System and the "Ambiguity Resolver"

RAG-HAR+ changes the game with two main tricks.

1. The Offline "Filing System" Designer
Before you even start your workout, the system uses the AI once to design a custom filing system for your specific dataset. Think of this as the AI acting as a librarian who studies your library and decides, "Okay, for this collection of books, I should organize them by color and author, not by genre and page count." The AI looks at thousands of different ways to describe a movement (like how fast it is, how bumpy it is, or its rhythm) and picks the best three groups of clues that work best for your specific sensors and activities. This happens only once, offline, so it doesn't slow you down later.

2. The "Ambiguity Resolver" (The Smart Fallback)
Now, when you are moving, the system doesn't call the AI immediately. Instead, it takes your current movement, translates it into those custom clues, and searches the database for the top 10 most similar past examples.

The Majority Vote: If 8 out of 10 of those past examples say "You are running," the system just says, "Okay, you're running," and moves on. No AI needed!
The Fallback: If the database is confused—say, 5 examples say "running" and 5 say "jogging"—then the system calls the AI. This AI, now called the "Ambiguity Resolver," looks at the confusing clues and the past examples to make the final, smart decision.

This means the expensive AI is only called for the tricky, confusing moments. For the easy, obvious moments, the system runs on its own, using simple math to count votes.

The Results: Speed, Savings, and Smarts

The researchers tested this new system on six different datasets, ranging from people walking and running to complex industrial assembly tasks. The results were impressive:

Accuracy: The new system was just as good, or even better, at recognizing activities than the old method. On some datasets, it improved accuracy significantly (for example, on the MHEALTH dataset, it went from 96.91% to 98.35% accuracy).
Cost Savings: Because it stopped calling the AI for every single sample, it reduced the amount of data sent to the AI by 89.3% to 99.9%. In one case (MHEALTH), it only had to call the AI for one single sample out of 666!
Speed: The system became much faster. On the MHEALTH dataset, the time it took to recognize an activity dropped from 18.25 seconds to just 0.41 seconds. That's a speed-up of over 44 times! Even on the slowest dataset, it was still about 5.7 times faster.

Why This Matters for the Future

The paper also built a working prototype on a real smartphone to show that this isn't just a theory. They turned it into a widget in a fitness app. When they tested it on a real phone, the speed-up was even more dramatic—about 15 times faster—because the phone didn't have to wait for the internet to talk to the AI for every single step.

The authors suggest that this approach is a big step toward making smart health and fitness apps that work everywhere, even without a perfect internet connection, and without costing a fortune to run. It proves that you don't need to ask the "genius" for help on every little thing; sometimes, a smart filing system and a simple vote are all you need. The paper concludes that by shifting the AI's role from a constant online worker to a smart designer and a backup expert, we can make activity recognition cheaper, faster, and ready for the real world.

Technical Summary: RAG-HAR+ Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deployment

Problem Statement

Human Activity Recognition (HAR) from wearable sensors is critical for healthcare, fitness, and smart environments. However, existing deep learning (DL) approaches face significant barriers to practical edge deployment: they require dataset-specific training, large labeled corpora, and suffer from performance degradation under domain shifts (e.g., changes in sensor placement or subject). While Large Language Models (LLMs) offer a training-free alternative with strong semantic reasoning, their direct application to HAR is hindered by the continuous, multi-channel nature of sensor data, which exhausts context budgets and incurs high computational costs and latency.

The authors' prior work, RAG-HAR, addressed this by framing HAR as a retrieval-augmented task, using statistical descriptions of sensor windows to retrieve similar labeled examples to guide LLM classification. However, RAG-HAR retained two critical limitations for continuous edge deployment:

Fixed Retrieval Representation: It used a static set of basic statistics (mean, max, min, etc.) across all datasets, failing to capture dataset-specific discriminative cues (e.g., periodicity for locomotion vs. magnitude for postures).
Mandatory LLM Inference: The LLM was invoked for every test sample, incurring network round-trips, high token costs, and variable latency, even when retrieved neighbors already provided a confident answer.

Methodology: RAG-HAR+

RAG-HAR+ introduces a retrieval-first, cost-optimized framework that restructures the LLM's role from a per-sample classifier to a specialized, conditional resource. The framework consists of two distinct stages:

1. Offline Retrieval Designer Agent

Before inference, an LLM-based agent acts as a Retrieval Designer to optimize the retrieval representation for a specific dataset.

Feature Pool: The agent is provided with a diverse pool of descriptors spanning statistical, temporal, spectral, motion-intensity, and signal-shape categories.
Iterative Design: Through an iterative process, the agent selects and partitions features into three compact, non-overlapping feature groups ( $G^*_1, G^*_2, G^*_3$ ).
Feedback Loop: The agent proposes groups, evaluates them on a validation set using Retrieval F1 (performance of majority voting on retrieved neighbors) and RAG Hit Rate (presence of the correct label in top- $q$ neighbors), and refines the groups via "improvement prompts" to address weak classes or redundancy.
Outcome: This results in dataset-specific, multi-vector representations that better align with the unique motion patterns of the target activity taxonomy.

2. Online Inference Pipeline

During real-time inference, the system operates as follows:

Multi-Vector Search: A query window is processed using the selected feature groups to generate three query vectors. These are searched against a multi-vector index (where each training sample is stored as three separate vector fields).
Weighted Re-ranking: Results from the three vector fields are merged and re-ranked using optimized weights to select the top- $q$ neighbors.
Deterministic Routing (Majority Voting): The system applies a parameter-free majority vote on the labels of the top- $q$ $q$ neighbors.
- Clear Majority: If a single label wins, the activity is predicted directly. No LLM is invoked.
- Ambiguity: If two or more labels tie for the highest vote, the sample is routed to the Ambiguity Resolver Agent.
Ambiguity Resolver Agent: For ambiguous cases, the LLM is invoked online. It receives a structured prompt containing the query window's statistical descriptions (computed over start, mid, end, and full segments) and the retrieved examples, allowing it to reason over the specific context to resolve the tie.

Key Contributions

Two-Stage LLM Usage: The paper reframes LLM usage in HAR into a one-time offline Retrieval Designer stage and a conditional online Ambiguity Resolver stage. This shifts the inference cost from scaling with the number of samples to scaling with prediction ambiguity.
Adaptive Feature Grouping: The introduction of an LLM-guided agent that constructs compact, dataset-specific multi-vector retrieval groups, replacing the fixed, generic statistical descriptors of previous methods.
Mobile Prototype: Implementation of a smartphone-based prototype (Flutter app) demonstrating the practical feasibility of this retrieval-first approach in a real-world mobile edge setting, including on-device latency measurements and incremental index updates.
Comprehensive Evaluation: An extended evaluation across six HAR benchmarks (USC-HAD, PAMAP2, MHEALTH, GOTOV, HHAR, Skoda) with ablation studies isolating the contributions of the retrieval designer, feature grouping, and routing strategies.

Results

Performance: RAG-HAR+ achieves competitive or improved performance compared to RAG-HAR and state-of-the-art non-LLM baselines. It improved F1 scores on four of the six datasets (USC-HAD, MHEALTH, HHAR, Skoda) while remaining competitive on the others.
Cost Reduction: The framework drastically reduces online LLM usage. Across the six datasets, online token consumption was reduced by 89.3% to 99.9%. For instance, on the MHEALTH dataset, only one sample required an LLM call out of 666 test samples.
Latency: Inference latency was reduced by 5.7x to 44.5x compared to RAG-HAR. On the mobile prototype, latency dropped from ~5.4 seconds to ~0.36 seconds per sample (a ~15x speed-up), primarily because most samples bypass the network round-trip to the LLM.
Retrieval Quality: The adaptive feature groups significantly improved the Retrieval F1 (performance of voting alone), narrowing the gap between retrieval-only predictions and final LLM-assisted predictions. This indicates that the retrieval step provides stronger evidence, reducing the need for fallback.

Significance and Claims

The paper claims that RAG-HAR+ demonstrates the practical feasibility of training-free, retrieval-augmented HAR for edge deployment. By decoupling the LLM from the critical path of every inference, the system achieves a scalable operating point where:

Inference cost scales with ambiguity, not data volume.
Retrieval quality is enhanced through dataset-specific, LLM-designed feature representations without fine-tuning a classifier.
Edge deployment is viable for continuous monitoring, as the system minimizes network dependency and token costs while maintaining high accuracy.

The authors position this work as a step toward scalable, generalizable HAR systems that leverage the reasoning capabilities of LLMs without incurring prohibitive computational or financial costs, making them suitable for resource-constrained wearable and mobile environments.