MuRAL: A Multi-Resident Ambient Sensor Dataset Annotated with Natural Language for Activities of Daily Living

The paper introduces MuRAL, a novel multi-resident ambient sensor dataset enriched with natural language annotations and fine-grained labels to bridge the gap between current Large Language Models and complex smart home activity understanding, while benchmarking their performance to highlight existing challenges in resident assignment and context integration.

The Gist

Imagine you are trying to teach a super-smart robot how to understand what's happening inside a house. You want the robot to know who is doing what, when they are cooking dinner, or when they are just relaxing on the couch.

For a long time, scientists have tried to do this using "smart home" sensors (like motion detectors or switches on doors). But there's a big problem: The data is too boring and confusing.

Think of the old data like a cryptic diary written by a robot that only speaks in code. It might say: "Door opened. Light turned on. Motion detected." It doesn't tell you who did it, why they did it, or if two people were doing it at the same time. It's like trying to guess the plot of a movie just by looking at a list of camera flashes.

Recently, we have Large Language Models (LLMs)—the same AI brains behind chatbots that can write stories and solve riddles. Scientists thought, "If we just talk to these AIs in plain English, they should be able to figure out the house's story!"

But there was a catch: We didn't have a good textbook to teach them. The old datasets were like puzzle pieces with missing pictures. They lacked the "story" (natural language) and the "characters" (who is who).

Enter MuRAL: The "Scripted Reality Show" Dataset

The authors of this paper created MuRAL (Multi-Resident Ambient sensor dataset with natural Language). Think of MuRAL as a behind-the-scenes script for a reality TV show, but filmed in a real smart apartment.

Here is what makes it special, using some simple analogies:

1. The Cast of Characters (Multi-Resident)
Old datasets were like a solo act—one person living alone. MuRAL is like a family reunion. They recorded 21 different sessions with 2 to 4 people living together. This is crucial because in real life, people bump into each other, talk over each other, and do things at the same time. It's chaotic, just like a real home.

2. The "Translator" (Natural Language)
Instead of just recording "Sensor 4: ON," the researchers watched video recordings of the sessions and wrote down exactly what happened in plain English.

• Old Data: "Motion detected in kitchen."
• MuRAL Data: "Person A opened the fridge, took out eggs, and walked to the counter."
  This is like giving the AI a commentary track to listen to while it watches the raw sensor data. It bridges the gap between cold numbers and human meaning.

3. The "Secret Handshake" (Privacy)
To keep things private, they didn't use cameras to record the final dataset. They only used cameras to help the human annotators write the "script" (the descriptions). Once the script was written, the video was deleted. It's like a director watching a rehearsal to write the script, then burning the rehearsal tape so no one sees the actors' faces, but keeping the script for the AI to learn from.

The Big Test: Can the AI Read the Room?

The researchers took the smartest AI models available (like GPT-4o) and gave them the raw sensor data from MuRAL, asking them to do three things:

1. Who is who? (Subject Assignment)
2. What are they doing? (Action Description)
3. What is the big picture? (Activity Classification)

The Results: The AI is smart, but still gets confused.

• The Good News: The AI could understand the data much better than before because of the English descriptions.
• The Bad News: The AI still struggles with long stories. If the session is long, the AI starts to forget who is who. It's like a student trying to remember the names of 4 people in a crowded room for an hour; eventually, they mix up who is holding the coffee cup and who is holding the book.
• The Tricky Part: The AI often mistakes "watching TV" for "sitting on the couch." It sees the person sitting and thinks, "Ah, resting!" It misses the context that they are actually playing a video game or watching a show. It needs to connect the dots over time, which is still very hard for it.

Why Does This Matter?

This paper is like handing a new, better textbook to the next generation of AI.

• For the Future: It shows us that to build truly smart homes that help elderly people or manage security, we need data that includes who is doing what in plain language.
• The Lesson: We can't just feed AI raw numbers anymore. We need to teach it the story of human life. MuRAL is the first major step in creating that storybook for smart homes.

In short: MuRAL is the Rosetta Stone for smart homes, translating the confusing language of sensors into the clear, human language that AI needs to truly understand our daily lives.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have shown promise in human activity recognition (HAR) using ambient sensor data, existing benchmark datasets (e.g., CASAS, ARAS, MARBLE) are insufficient for leveraging modern LLM capabilities. These traditional datasets suffer from three main limitations:

• Lack of Natural Language Context: They rely on high-level activity labels rather than fine-grained textual descriptions, preventing the use of LLMs' zero-shot reasoning and in-context learning abilities.
• Simplistic Scenarios: They often feature scripted routines in single-resident settings, failing to capture the complexity, ambiguity, and social dynamics of real-world multi-resident environments.
• Discrete Sensor Data: Ambient sensors (e.g., door contacts, motion detectors) output low-level, discrete states (open/closed) that lack semantic meaning without rich context, making interpretation difficult for conventional models.

The paper addresses the gap between the potential of LLMs for semantic reasoning and the lack of suitable, richly annotated datasets for multi-resident smart homes.

2. Methodology

A. Dataset Collection (MuRAL)

• Environment: Data was collected in the DOMUS Intelligent Apartment (60 m²), equipped with 23 ambient sensors (6 motion, 10 magnetic contacts, 7 smart plugs). Sensors were non-overlapping and assigned to specific zones.
• Participants: 18 volunteers participated in 21 sessions (totaling >21 hours of data). Sessions involved 2 to 4 residents acting in roles (e.g., family, roommates) during specific timeframes (weekday/weekend).
• Protocol: Unlike scripted datasets, participants were given high-level context (roles, time of day) and a list of possible activities but were free to act spontaneously. This ensured naturalistic, overlapping, and dynamic interactions.
• Privacy Measures: No personal identifiers were recorded. Video/audio were used only for annotation and permanently deleted afterward. Bedroom activities were simulated (no sensors inside) to protect privacy.

B. Annotation Strategy

MuRAL introduces a novel, multi-level annotation scheme:

1. Subject Assignment: Residents are assigned alphabetical IDs (A, B, C...) based on the order of their first appearance in a session.
2. Fine-Grained Action Description: Annotators provided natural language descriptions for approximately 50% of sensor events (those representing significant changes).
   • Example: Instead of just "Kitchen Cabinet Closed," the annotation reads: "A takes out dishes from the kitchen cabinet, moves them to the dining room, and places the dishes on the table."
3. High-Level Activity Labels: Events are grouped by resident and mapped to 27 predefined activity categories (e.g., "preparing breakfast," "watching TV," "chatting").

C. Experimental Benchmarking

The authors evaluated state-of-the-art LLMs (GPT-4o and GPT-4o-mini) on MuRAL using a two-stage pipeline:

1. Stage 1 (Event Processing): The LLM processes blocks of 10 sensor events to assign resident IDs, generate action descriptions, and update resident states.
2. Stage 2 (Activity Classification): The generated action descriptions are fed into a second LLM pass to infer high-level activity labels.

• Baselines: The proposed approach was compared against LAHAR, a previous state-of-the-art method that merges adjacent sensor events. The authors modified the pipeline to treat events individually to align with MuRAL's fine-grained annotations.

3. Key Contributions

• MuRAL Dataset: The first multi-resident ambient sensor dataset featuring fine-grained natural language descriptions for sensor events, explicit resident identities, and rich contextual metadata.
• Realistic Multi-Resident Scenarios: Captures complex social dynamics, overlapping activities, and temporal transitions involving 2–4 people, addressing a gap left by single-resident datasets.
• Comprehensive Benchmarking: Establishes a baseline for LLMs in smart home HAR, evaluating three core tasks: Subject Assignment, Action Description, and Activity Classification.
• Public Availability: The dataset and code are publicly released to foster research in language-driven and socially aware HAR.

4. Results

The evaluation revealed that while LLMs can interpret sensor data, significant challenges remain:

• Subject Assignment:
  • Accuracy declines as the sequence length (window size) increases, indicating error propagation in long sequences.
  • GPT-4o outperformed GPT-4o-mini.
  • The LAHAR preprocessing (merging events) helped the smaller model (4o-mini) but degraded performance for the stronger model (4o) in complex multi-resident scenarios, as merging events from the same sensor often conflated different residents.
• Action Description:
  • The authors' method (event-level processing) outperformed LAHAR in semantic similarity to ground truth.
  • GPT-4o-mini surprisingly produced descriptions slightly closer to ground truth than 4o, likely due to stronger instruction-following capabilities and fewer omissions.
  • Overall, generating accurate descriptions from raw sensor data remains highly challenging.
• Activity Classification:
  • Neither approach achieved high zero-shot accuracy.
  • Major Confusion: Activities requiring temporal integration (e.g., "playing video games," "watching TV") were frequently misclassified as "resting." Models failed to link a sequence of actions (turning on console + sitting down) to a single coherent activity or to distinguish between a resident resting vs. watching TV with others.

5. Significance

• Advancing HAR: MuRAL shifts the paradigm from simple classification to semantic reasoning, enabling the development of systems that understand why an action occurred, not just what happened.
• LLM Evaluation: It provides a rigorous testbed for evaluating the reasoning capabilities of LLMs in physical, multi-agent environments, highlighting current limitations in long-term memory and context integration.
• Future Research: The dataset paves the way for next-generation intelligent systems that can handle ambiguity, social interaction, and complex household routines, moving beyond scripted, single-user scenarios.
• Ethical Standards: The dataset adheres to strict GDPR and ethical guidelines, offering a privacy-preserving model for future smart home research.