🤖 AI

RAG-HAR: Retrieval Augmented Generation-based Human Activity Recognition

RAG-HAR is a training-free, retrieval-augmented framework that leverages large language models and optimized prompt engineering to achieve state-of-the-art performance in human activity recognition across diverse benchmarks while enabling the identification of unseen activities without requiring model fine-tuning or large labeled datasets.

Original authors: Nirhoshan Sivaroopan, Hansi Karunarathna, Chamara Madarasingha, Anura Jayasumana, Kanchana Thilakarathna

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nirhoshan Sivaroopan, Hansi Karunarathna, Chamara Madarasingha, Anura Jayasumana, Kanchana Thilakarathna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a smartwatch that tracks your movements. Usually, to teach a computer what you are doing (like walking, running, or sitting), you have to build a custom "teacher" for every single person and every new activity. You feed it thousands of hours of data, it studies hard, and then it gets good at guessing. But if you change the watch, the person, or the activity, the teacher gets confused and you have to start over.

RAG-HAR is a new way to do this that skips the "studying" part entirely. Instead of training a new teacher, it uses a super-smart, pre-existing "expert" (a Large Language Model, or LLM) and gives it a cheat sheet right before it makes a guess.

Here is how it works, using simple analogies:

1. The Problem: The "Blank Slate" Expert

Think of a standard AI as a student who has read every book in the world but has never seen a specific type of motion before. If you show it a sensor reading from a new person running, it might guess "walking" because it doesn't have a specific reference for this person's run. It's like asking a chef to cook a dish they've never seen, just by describing the ingredients.

2. The Solution: The "Library" Approach (RAG)

RAG-HAR changes the game. Instead of asking the expert to guess from memory, it gives them a library of examples right next to them.

The Cheat Sheet (Vector Database): Before the AI makes a guess, the system looks at the current movement (like a 2-second clip of you moving). It breaks this movement down into simple math facts (like "how fast was the average speed?" or "how much did it shake?"). It then goes to a giant digital library and finds the 10 most similar past examples.
The Context: It hands these 10 examples to the expert AI and says, "Look, this new movement looks a lot like these 10 things. Based on that, what is it?"

3. The Two-Step Process

The paper describes this happening in two main phases:

Phase 1: The Baseline (The Quick Look)
The system takes your movement, turns it into a list of numbers (statistics), and finds similar lists in the library. It then asks the AI: "Here are 10 similar past movements and their labels. What is this new one?"
- Result: This alone is already better than many complex, trained systems because the AI can "reason" using real examples instead of just guessing.
Phase 2: The Optimization (The Expert Polish)
The researchers realized the AI could do even better if the "cheat sheet" was written more clearly and the questions were asked more cleverly.
- Better Descriptions: Instead of just giving the AI raw numbers, they use the AI itself to write a short, natural language story about the movement (e.g., "This looks like a rhythmic, high-intensity motion with a steady beat"). This helps the AI understand the feeling of the movement, not just the math.
- Prompt Engineering: They used a special tool to automatically write the perfect instructions for the AI, testing thousands of different ways to ask the question until they found the one that got the best answers.

4. Why This is a Big Deal

The paper highlights three major superpowers of RAG-HAR:

No Training Required: You don't need to spend weeks teaching the AI. You just add new examples to the library. If you want the AI to recognize "dancing," you just add a few dancing examples to the library. The AI instantly knows how to do it.
It Can Guess the Unknown: Traditional AI gets stuck if it sees something it wasn't trained on (like a new type of exercise). RAG-HAR can look at a weird new movement, say, "I haven't seen this exact thing, but it looks like a mix of jogging and jumping," and give it a meaningful name. It doesn't just say "I don't know."
It's Cheap and Fast: Because it doesn't need a massive computer to "train" the model, it saves a lot of money and time. It's like hiring a consultant who already knows everything, rather than hiring a student and spending years teaching them.

Summary Analogy

Imagine you are trying to identify a bird you've never seen.

Old Way (Deep Learning): You spend 10 years memorizing pictures of 50 specific birds. If you see a bird that isn't on your list, you fail.
RAG-HAR Way: You have a friend who is an expert birdwatcher. You show them the bird, and you also hand them a book of the 10 birds that look most like it. Your friend looks at the book, compares it to the bird in your hand, and says, "Ah, this looks like a mix of a sparrow and a finch, but it's definitely a new type of sparrow."

The paper proves that this "consultant with a reference book" approach works better than the "memorized student" approach for recognizing human activities, without needing any extra training.

Technical Summary: RAG-HAR

Problem Statement

Human Activity Recognition (HAR) from wearable sensor data is critical for healthcare, rehabilitation, and smart environments. However, existing deep learning (DL) approaches face three significant limitations:

Resource Intensity: They require costly, time-consuming, dataset-specific training and fine-tuning.
Generalization Issues: Performance degrades under domain shifts (e.g., different subjects, sensor placements, or devices).
Data Dependence: They rely heavily on large, labeled corpora.

Furthermore, while Large Language Models (LLMs) offer broad world knowledge and reasoning capabilities, direct application to sensor data often fails. LLMs struggle with fine-grained activity distinctions and lack grounding in specific activity patterns, leading to misclassifications when presented with raw or poorly contextualized sensor signals.

Methodology: RAG-HAR

The paper introduces RAG-HAR, a training-free framework that leverages Retrieval-Augmented Generation (RAG) to enable LLMs to perform HAR without model training or fine-tuning. The framework operates in two stages: a Baseline stage and an Optimization stage.

1. Data Processing and Feature Extraction

Normalization: Raw sensor channels (e.g., accelerometer, gyroscope, magnetometer) are standardized using Z-score normalization.
Sliding Window & Partitioning: Time-series data is segmented into fixed-size windows. Crucially, each window is further partitioned into four sub-segments: the Full segment, and three equal parts (Start, Mid, End).
Statistical Descriptors: For each segment and sub-segment, eight statistical features are computed across all channels: mean, max, min, 25th/75th percentiles, standard deviation, median, and peak count. This captures both global trends and local transient events.

2. Baseline Stage (Retrieval-Augmented Reasoning)

Embedding Generation: The statistical feature vectors are serialized into structured text strings (e.g., mean=0.12, std=0.34). These strings are embedded using a pre-trained, general-purpose text embedding model (e.g., text-embedding-3-small).
Vector Database Indexing: Embeddings from the training data are stored in a vector database along with metadata (activity labels, user IDs).
Inference & Retrieval:
- For a test sample, embeddings are generated for its segments and sub-segments.
- An Approximate Nearest Neighbor (ANN) search retrieves the top- $k$ most similar training samples based on cosine similarity.
- A weighted re-ranking mechanism combines scores from the different sub-segments to select the final context set.
LLM Classification: The retrieved samples (with their labels and statistical features) and the query sample are formatted into a prompt. A pre-trained LLM (e.g., gpt-5-mini) analyzes the contextual evidence and the query to predict the activity label and provide a rationale.

3. Optimization Stage

To further enhance performance, two strategies are introduced:

Prompt Optimization: A meta-optimization loop generates and iteratively refines system prompts. Using a roulette-wheel selection strategy based on validation F1-scores, the system explores diverse prompt phrasings and strategies (e.g., specific feature extraction instructions) to maximize classification accuracy for a given dataset.
LLM-Based Activity Descriptors: Instead of using simple statistical templates for embedding, a specialized LLM prompt generates rich, natural language descriptions of the motion patterns (e.g., "sustained, repetitive motion," "dominant axis X"). These semantically richer descriptions are then embedded and indexed, improving the quality of retrieved context.

Key Contributions

Training-Free SOTA Performance: RAG-HAR is the first retrieval-augmented framework for HAR that achieves state-of-the-art (SOTA) performance across six diverse benchmarks without any dataset-specific model training or fine-tuning.
Two-Stage Framework: The paper proposes a novel architecture combining a baseline retrieval system with an optimization stage featuring prompt optimization and LLM-based activity descriptors, offering a controllable trade-off between efficiency and accuracy.
Open-World Recognition: Unlike traditional DL models constrained to fixed taxonomies, RAG-HAR can detect and generate meaningful labels for unseen activities. It moves beyond collapsing unknown inputs into a single "unknown" class, instead utilizing the LLM's semantic knowledge to categorize novel behaviors.
Comprehensive Evaluation: The framework is validated on six datasets (HHAR, PAMAP2, MHEALTH, GOTOV, SKODA, USC-HAD) covering 6 to 60 sensor channels and various activity types (locomotion, daily living, assembly tasks).

Results

Benchmark Performance: RAG-HAR outperforms leading DL baselines (including CNNs, LSTMs, and Transformers) across five of the six datasets.
- On PAMAP2, it achieved an F1-score of 91.12%, surpassing the previous best (Triplet LSTM) by ~0.5%.
- On MHEALTH, it reached 96.74%, outperforming adversarial feature extractors.
- On GOTOV and SKODA, it improved F1-scores by 1–2% over the strongest prior models.
- On HHAR, it showed a significant improvement of over 7% F1-score compared to SimCLR and DeepConvLSTM baselines.
Open-Set Capability: In open-set experiments (where 30–70% of activity classes are withheld from the database), RAG-HAR consistently outperformed the MTMD baseline by 3–7% in F1-score.
Unseen Activity Labeling: The system demonstrated the ability to generate semantically coherent labels for novel activities. For single unknown classes, it achieved 96.47% accuracy in mapping predicted labels to ground truth; this remained robust (88.90%) even with three unknown classes withheld.
Cost and Efficiency: The framework eliminates training time. Inference costs are minimal (approx. $0.0006 per sample on MHEALTH), and the system avoids the need for GPU-intensive training pipelines.

Significance and Claims

The paper claims that RAG-HAR fundamentally shifts the HAR paradigm from model-centric learning to retrieval-augmented reasoning. Its primary significance lies in:

Eliminating Training Overhead: It removes the need for expensive, dataset-specific training cycles, making HAR more accessible and scalable.
Handling Unseen Data: It addresses the "open-world" problem by enabling the recognition and meaningful labeling of activities not present in the training data, a capability traditionally requiring extensive human annotation.
Robustness: By grounding predictions in retrieved examples and leveraging the LLM's pre-trained world knowledge, the system achieves high generalization across different subjects and sensor configurations without retraining.
Practicality: The framework offers a lightweight, cost-effective alternative to deep learning, suitable for deployment in resource-constrained environments where retraining is impractical.