ORACLE: Orchestrate NPC Daily Activities using Contrastive Learning with Transformer-CVAE

The paper introduces ORACLE, a novel generative model that combines Transformers, Conditional Variational Autoencoders, and contrastive learning to synthesize realistic and diverse daily activity plans for Non-player characters (NPCs) by addressing data imbalance and scarcity challenges in smart home datasets.

The Gist

Imagine you are walking through a video game world. You see a character sitting on a bench, then suddenly they teleport to a kitchen, then instantly to a park, then back to the bench. It feels weird, right? That's because most computer characters (NPCs) don't actually "live" a life; they just follow a broken script.

This paper introduces ORACLE, a new AI system designed to fix this. Think of ORACLE as a super-smart life coach for video game characters. Its job is to write a believable 24-hour schedule for an NPC, just like a real human would have one.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Broken Record" Effect

Old ways of making NPC schedules are like a broken record player. They either:

• Follow strict rules: "If it's 8 AM, eat breakfast. If it's 9 AM, go to work." This is boring and predictable.
• Guess randomly: They might try to eat breakfast at 3 AM or sleep for 14 hours straight.
• Get stuck: They can't handle long days (24 hours) well because their memory is short.

2. The Solution: ORACLE's "Three-Layer Cake"

The researchers built ORACLE using three special ingredients to bake a perfect daily schedule:

🍰 Layer 1: The Transformer (The "Super-Memory")

Imagine trying to remember what you did 10 hours ago while planning your next move. Old AI models get confused and forget.

• The Analogy: ORACLE uses a Transformer, which is like a librarian with a photographic memory. It can look at the entire 24-hour day at once, not just the last few minutes. It understands that if you just woke up, you probably need to brush your teeth before you cook breakfast. It connects the dots across the whole day.

🍰 Layer 2: The CVAE (The "Creative Director")

If you ask a robot to "plan a day," it might give you the exact same plan every time because it's trying to be safe.

• The Analogy: ORACLE uses a CVAE (Conditional Variational Autoencoder). Think of this as a Creative Director who adds a little bit of "chaos" or "flavor" to the plan.
  • Sometimes the NPC is tired and naps.
  • Sometimes they are energetic and go for a run.
  • The system knows the rules of a day (you can't sleep for 20 hours), but it allows for variety within those rules. It makes every NPC feel like a unique person, not a clone.

🍰 Layer 3: Contrastive Learning (The "Strict Editor")

This is the secret sauce. Sometimes the AI might generate a weird plan, like "Eat dinner, then immediately sleep, then wake up and eat breakfast again."

• The Analogy: ORACLE has a Strict Editor (using Contrastive Learning).
  • The AI generates a plan.
  • The Editor looks at it and says, "Wait, that doesn't make sense. Humans don't do that."
  • The Editor then shows the AI a "Good Example" (a real human schedule) and a "Bad Example" (the weird one).
  • The AI learns to push the "Bad" plans away and pull the "Good" plans closer. Over time, it learns to spot the difference between a realistic day and a crazy one.

3. The Training Data: The "Smart Home" Library

To learn how humans actually live, ORACLE didn't just guess. It studied a massive library of data called CASAS.

• The Analogy: Imagine a smart house where sensors track exactly what real people do all day—when they wake up, how long they cook, when they take a shower. ORACLE read thousands of these "diaries" to learn the rhythm of human life.
• The Challenge: The data was messy. Some people didn't label their activities, and some activities happened way more often than others (like sleeping vs. exercising). ORACLE had to clean this data up, grouping similar things together (like "cooking breakfast" and "cooking dinner" both becoming just "Cooking").

4. What Did They Find?

The researchers tested ORACLE against other AI models and even asked real humans to judge the schedules.

• The Result: ORACLE won.
• Why? The schedules it made were more diverse (less repetitive), more realistic (followed human logic), and more flexible (it could fill in the blanks if you told it, "This character must be at work at 9 AM," and it would figure out the rest of the day around that).

The Big Picture

Why does this matter?

• For Gamers: It means video game worlds will feel alive. NPCs will have their own lives, making the world feel less like a stage and more like a real place.
• For Real Life: This technology could help design better smart homes, plan cities, or even create digital twins for elderly care, predicting what a person might need before they ask.

In short: ORACLE is an AI that learned to stop acting like a robot and start acting like a human, by reading thousands of real diaries, using a super-memory to plan ahead, and having a strict editor to keep the plans sensible.

Technical Summary

1. Problem Statement

The paper addresses the challenge of generating realistic, diverse, and contextually coherent daily activity plans for Non-Player Characters (NPCs) in digital environments (e.g., video games, VR). Existing methods face three primary limitations:

• Long-term Dependency Issues: Traditional recurrent models (LSTMs/GRUs) struggle with vanishing gradients over long sequences (24-hour cycles), leading to short-term planning that lacks temporal coherence.
• Lack of Diversity: Deterministic models often produce repetitive, monotonous outputs. They optimize for likelihood maximization, biasing them toward frequent patterns rather than exploring plausible but less common variations.
• Constraint Integration: It is difficult to integrate partially defined constraints (e.g., "NPC must work from 9 AM to 5 PM") while maintaining natural transitions between activities.
• Data Scarcity & Cost: While Large Language Models (LLMs) can plan behaviors, they are computationally expensive, have high inference latency, and lack training on specific 24-hour routine corpora, often yielding generic schedules.

2. Methodology: The ORACLE Framework

The authors propose ORACLE, a generative framework combining Transformers, Conditional Variational Autoencoders (CVAE), and Contrastive Learning. The system is trained on the CASAS smart home dataset, which contains 24-hour indoor activity sequences.

A. Data Preprocessing & Filtering

• Dataset: Utilized the CASAS Apartment (42 classes) and Home (46 classes) datasets.
• Consolidation: Merged 42–46 activity classes into 12 major categories (e.g., Sleep, Outing, Work, Hygiene, Meal) to mitigate class imbalance and semantic fragmentation.
• Filtering: Applied strict criteria based on duration and frequency (derived from ChatGPT knowledge, expert intuition, and user surveys) to remove unrealistic sequences (e.g., sleeping >12 hours or skipping meals).
• Tokenization: Converted 24-hour sequences into 288 discrete tokens (5-minute intervals).

B. Core Architecture: Transformer-CVAE

• Encoder: Uses a Transformer encoder to process activity tokens ($A$) and condition tokens ($M$, representing masked/unmasked states). It employs self-attention to capture global dependencies.
• Latent Space: The encoder output is mean-pooled and projected into a latent distribution defined by mean ($\mu$) and variance ($\sigma$). A reparameterization trick samples a latent vector $z$.
• Decoder: A Transformer decoder generates the sequence autoregressively. It takes the latent vector $z$ and the condition tokens $M$ (for partial schedule augmentation) as input to predict the next activity token.
• Loss Function (Reconstruction): Optimized via Maximum Likelihood Estimation (MLE) to minimize negative log-likelihood ($L_{recon}$).

C. Contrastive Learning Mechanism

To address the "plausibility" gap and exposure bias, the authors introduce a novel contrastive learning objective:

• Positive Samples ($A_{pos}$): Generated sequences that satisfy the predefined plausibility criteria (Table 1).
• Negative Samples ($A_{neg}$): Generated sequences that violate these criteria (e.g., illogical time gaps or impossible activity durations).
• Objective: The model is trained to minimize the distance between the generated sequence and positive samples while maximizing the distance from negative samples. This acts as a "weak supervision" mechanism to refine the generative process without requiring pre-trained models.
• Loss: $L_{contrastive} = (1 - \text{sim}(A, A_{pos})) + \text{sim}(A, A_{neg})^2$.

3. Key Contributions

1. Novel Generative Model: Introduced ORACLE, the first framework to integrate Transformer-CVAE with contrastive learning specifically for 24-hour NPC activity synthesis.
2. Flexible Generation: Supports both unconditional generation (creating a full day from scratch) and conditional generation (augmenting or completing partially defined schedules).
3. Contrastive Refinement: Proposed a contrastive learning strategy tailored for discrete, long-horizon sequence generation to filter out unrealistic "near-miss" outputs, improving coherence without heavy pre-training.
4. Comprehensive Evaluation: Validated the model through quantitative metrics, ablation studies, and a user study, demonstrating superiority over LSTM and standard Transformer baselines.

4. Experimental Results

The model was evaluated on the CASAS Apartment and Home datasets against baselines: Bowman (LSTM-VAE), Vaswani (Transformer), and Koushik (BiLSTM).

Quantitative Metrics

• Random Generation: ORACLE achieved the lowest Wasserstein Distance (WD) (0.4835 vs. 0.6263 for Bowman), indicating its output distribution closely matches real human data. It also scored highest on LLM Score (plausibility) and Distinct-10/15 (diversity), proving it avoids mode collapse.
• Conditional Generation: While ORACLE had a slightly higher Sequence Alignment Method (SAM) distance than BiLSTM (due to its generative diversity), it significantly outperformed other models when the masking ratio was high (>90%). It achieved the highest REAL-Condition scores, meaning it generated schedules that were both constraint-compliant and diverse.
• Memorization Check: k-NN analysis confirmed ORACLE does not simply copy training data; generated sequences had high Hamming mismatch rates (30–40% difference) compared to the nearest training neighbor.

User Study & Ablation

• User Perception: In a study with 11 participants, ORACLE received higher ratings for Naturalness, Diversity, Feasibility, and Sustainability compared to all baselines and even compared to general-purpose LLMs.
• Ablation Study: Removing contrastive learning (ORACLE-C) or replacing the Transformer with an LSTM (ORACLE-T) resulted in significant performance drops. Visualizations of latent spaces and attention maps confirmed that contrastive learning reduces bias and encourages the model to focus on diverse, relevant features.

5. Significance and Future Work

• Significance: ORACLE provides a computationally efficient, domain-specific solution for NPC behavior planning that balances realism (adhering to human constraints) and diversity (avoiding repetition). It bridges the gap between rigid rule-based systems and expensive LLMs.
• Applications: The framework is applicable to smart home automation, elderly care monitoring, urban planning simulation, and immersive gaming/VR.
• Limitations & Future Directions: The current model is limited to generalized behaviors inferred from indoor smart home data and cannot yet handle highly personalized routines or outdoor environments. Future work aims to integrate reinforcement learning to simulate broader decision-making processes across varied environments.