⚡ electrical engineering

A General Set-Based Framework for Cognitive State Estimation: Theory and Application to Conditionally Automated Driving

This paper introduces a set-based framework that estimates human cognitive states (trust, perceived risk, and workload) in conditionally automated driving by treating uncertainties as unknown but bounded, demonstrating through a simulator experiment that it outperforms probabilistic baselines in predicting human reliance without requiring large datasets or distributional assumptions.

Original authors: Sibibalan Jeevanandam, Neera Jain

Published 2026-08-04

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Sibibalan Jeevanandam, Neera Jain

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to drive a car that can steer itself, but only if you are paying attention. This is the world of "Level 3" automated driving: the car does the heavy lifting, but the human driver is the safety net, ready to grab the wheel if things get weird. The big problem? Humans are tricky. Sometimes we trust the car too much and zone out; other times, we panic and take over too early. To make these cars safe, the computer needs to know what's going on inside the driver's head—specifically, how much they trust the car, how risky they think the situation is, and how stressed they feel. This field of study is called Human-Automation Interaction. For a long time, scientists have tried to guess these mental states using math that assumes human behavior follows a predictable "bell curve" of randomness, like rolling dice. But humans aren't dice; we are messy, and we don't always have enough data to roll the dice enough times to know the pattern.

This paper introduces a clever new way to guess what a driver is thinking without needing to know the "rules of the dice." Instead of guessing probabilities, the researchers use a "set-based" approach. Think of it like a game of "Hot or Cold" played with a shrinking box. Instead of saying, "The driver is 70% likely to be stressed," the system says, "The driver's stress level is definitely somewhere inside this specific box." As the driver interacts with the car, the box gets smaller and more precise. The researchers tested this in a driving simulator with 20 volunteers, asking them to drive while the car took over. They found that this "shrinking box" method was better at predicting when a driver would take over the wheel than the traditional "dice-rolling" methods, especially when looking further into the future.

The Paper's Story: A Detective with a Shrinking Box

The Problem: The "Dice" Don't Roll Right
Most previous attempts to read a driver's mind rely on probability. They assume that if a driver is stressed, it's like rolling a die where "stress" comes up 30% of the time. To make this work, you need a massive amount of data to figure out exactly how the die is weighted. But in real life, you don't have millions of driving trials for every single person. Plus, human thoughts don't always follow neat, smooth curves. Sometimes, a driver might be perfectly calm until a construction zone appears, and then suddenly, they are on high alert. The old methods struggle with these sudden jumps and the lack of data.

The New Idea: Bounding the Unknown
The authors, Sibibalan Jeevanandam and Neera Jain, propose a different strategy. Instead of asking, "What is the chance the driver is stressed?", they ask, "What is the range of possibilities that fits what we see?"

Imagine you are trying to find a lost cat in a dark house.

The Old Way (Probabilistic): You guess the cat is 80% likely to be in the kitchen, 15% in the bedroom, and 5% in the garage. You need a lot of past data about where cats usually hide to make these guesses.
The New Way (Set-Based): You don't guess probabilities. Instead, you draw a box around the kitchen. You say, "We know for a fact the cat is somewhere inside this box." As you hear a meow or see a paw print, you shrink the box. If the cat is definitely in the kitchen, the box shrinks to just the kitchen. If you hear a meow from the hallway, the box expands to include the hallway. You never need to know the "probability" of the cat being there; you just need to know the box is big enough to catch the cat, but small enough to be useful.

In this paper, the "cat" is the driver's mental state (Trust, Perceived Risk, and Workload). The "box" is a mathematical set that contains all possible values for these states. The researchers treat the unknown factors (like sudden distractions or unmeasured thoughts) as "unknown but bounded." They don't know exactly how much the driver's mind wanders, but they know it won't wander off the planet. So, they draw a box big enough to cover that wander, and then they shrink it as they get more clues.

How the Detective Works
The system acts like a detective solving a mystery in two ways:

The "Self-Report" Clue: Every so often, the driver is asked, "How much do you trust the car?" or "How risky does this feel?" The driver answers on a scale (like 0 to 100). This is a bit fuzzy because the driver might round their answer or be slightly biased. The system takes this answer and draws a "feasible set" (a box) around it.
The "Reliance" Clue: The most important clue is what the driver does. Did they let the car drive (reliance = 1), or did they take the wheel (reliance = 0)? This is a binary clue: yes or no. The system uses a special map (a decision tree) to figure out which mental states lead to which action. For example, the map might say, "If Trust is high AND Risk is low, the driver will rely on the car."

By combining these clues, the system updates its box. If the driver says "I trust the car a lot" (Self-Report) but then immediately takes the wheel (Reliance), the system knows something is wrong with its current box. It shrinks the box to find a new spot where both clues make sense. If the box ever becomes empty (meaning no possible mental state fits the clues), the system realizes its map is wrong and redraws the whole map using the new data.

The Experiment: The Simulator Test
To test this, the researchers put 20 people in a driving simulator. The car was set to Level 3 automation, meaning it could drive itself, but the driver had to be ready to take over. The route had "construction zones" (high complexity) and normal roads (low complexity).

The drivers had to decide whether to let the car drive or drive themselves.
Every few minutes, they stopped at a red light and answered questions about their trust, risk, and workload.
The researchers used the first half of the drive to "train" the system (draw the initial maps and boxes) and the second half to test if the system could guess the driver's state in real-time.

What They Found
The results were promising. The "shrinking box" method worked well for most people:

Consistency: For about 80% of the participants, the system's box successfully contained the driver's actual mental state at least 75% of the time. For four participants, it was 100% consistent.
Speed: The system was fast, calculating new boxes in about 0.03 to 0.07 seconds, which is much faster than the 1-second intervals between data points. This means it could run in real-time on a real car.
Prediction Power: The biggest win was predicting the future. The researchers asked, "Can we guess if the driver will take over the wheel 15, 30, 45, or 60 seconds from now?"
- The new set-based method was the best predictor.
- As the prediction time got longer (e.g., 60 seconds ahead), the new method got better relative to the old methods.
- The traditional "particle filter" (the probabilistic method) got worse over time. The authors suggest this is because the old method's assumptions about "noise" (randomness) piled up errors over time, while the set-based method's "boxes" stayed tight and accurate by using the history of the driver's actions to constrain the possibilities.

Why It Matters
This paper suggests that we don't need to know the exact "probability" of a human's thoughts to build safe automated cars. We just need to know the boundaries of those thoughts. By using a set-based framework, the system can adapt to individual drivers, handle messy data, and predict future behavior more accurately than current methods. It's a step toward cars that don't just drive themselves, but actually understand the human sitting next to them, adjusting their behavior to keep everyone safe. The authors note that while this works well in a simulator, the next step is to see if it can handle the chaos of real-world driving and be used to automatically adjust the car's automation level to keep the driver engaged.

Technical Summary: A General Set-Based Framework for Cognitive State Estimation

Problem Statement
The paper addresses the challenge of estimating human cognitive states in Human-Automation Interaction (HAI) contexts, specifically within conditionally automated (SAE Level 3) driving. Current safety-critical automation requires drivers to supervise the system, yet drivers often suffer from either over-monitoring (high workload) or over-trust (neglecting alertness). Existing solutions for adaptive automation rely on real-time cognitive state estimation, but dominant approaches in the literature are probabilistic (e.g., Kalman filters, Particle filters, POMDPs). These methods require:

Large, structured datasets to identify noise distributions (typically Gaussian).
Frequent, periodic measurements, whereas real-world driving involves continuous interaction with sparse, asynchronous self-reports.
Often focus solely on "trust," neglecting other critical factors like perceived risk and workload.

The authors argue that human cognition is influenced by numerous unmeasured factors, making a probabilistic characterization of uncertainty less appropriate than a bounded, set-based characterization.

Methodology
The authors propose a set-based framework that treats process and measurement uncertainties as "unknown but bounded" rather than probabilistic. The framework operates in two phases: System Identification and Set-Valued State Estimation.

Modeling:
- State Space: The cognitive state vector $x$ includes Trust ( $T$ ), Perceived Risk ( $R$ ), and Workload ( $W$ ), all continuous-valued in $[0, 1]$ .
- Process Model: A bounded affine difference equation describes state evolution: $x^+(k) = Ax(k) + Bu(k) + C + w(k)$ , where $u$ represents task complexity and $w$ is unknown but bounded process noise.
- Measurement Models:
  - Self-Reports: Intermittent, quantized measurements (e.g., 0–100 scale) modeled as $y(k) = Q(x(k) + z(k))$ , where $z$ is bounded measurement noise.
  - Reliance: Binary observations ( $q \in \{0, 1\}$ ) indicating whether the driver uses the automation, modeled via a piecewise constant function $h(x)$ .
System Identification (Training Phase):
- Using a training segment of data, the authors estimate system matrices ( $A, B, C$ ) by minimizing prediction errors against self-reports.
- Noise Bounds: Instead of assuming a distribution, noise bounds ( $\epsilon_w, \epsilon_v$ ) are estimated by enforcing reachset conformance. This ensures that the set of states reachable by the model under bounded noise contains all observed self-reports.
- Reliance Regions: A binary decision tree is fitted to identify which cognitive states influence reliance and to partition the state space into regions ( $S_i$ ) with associated probabilities.
- Subspace Identification: The decision tree splits identify a specific subspace of cognitive states that actually influence reliance for a given individual, allowing for dimensionality reduction.
Set-Valued State Estimation (Testing Phase):
- The estimator maintains a set-valued estimate $X(k|k)$ of the cognitive states.
- Prediction: The set is propagated forward one step using the identified model and bounded noise.
- Update via Reliance: Upon observing binary reliance ( $q$ ), the predicted set is intersected with the feasible state set for that reliance observation ( $X^R_q$ ). To improve robustness, the estimator can process a block of past reliance observations.
- Update via Self-Reports: When a quantized self-report arrives, the set is intersected with the feasible set derived from the report.
- Online Adaptation: If the intersection of sets becomes empty (indicating the model fails to contain the true state), the system triggers a re-estimation of parameters using data up to that point.

Key Contributions

First Set-Based Approach: This is the first framework to apply set-valued state estimation to human cognitive states in a continuous, non-trial-based HAI interaction.
Robustness to Data Scarcity: The framework avoids the need for large datasets to characterize noise distributions, relying instead on bounded uncertainty and reachset conformance.
Multi-State Estimation: It simultaneously estimates Trust, Perceived Risk, and Workload, whereas prior work often focused only on trust.
Individualized Subspace Identification: The method identifies which specific cognitive states drive an individual's reliance behavior, enabling targeted estimation.
Online Adaptability: The inclusion of an online model update mechanism allows the system to adapt when initial parameter estimates fail to capture new behaviors.

Experimental Results
The framework was evaluated using data from 20 participants in a medium-fidelity driving simulator with SAE Level 3 automation.

Consistency: For 11 out of 15 analyzed participants, the set-valued estimator achieved at least 75% consistency (the estimated set overlapped with the true state implied by self-reports). Four participants achieved 100% consistency.
Uncertainty: The average uncertainty (volume of the estimated set relative to the state space) ranged from 0.42% to 38.37%, with 10 participants achieving uncertainty below 15%.
Prediction Performance: The authors evaluated $N_p$ $N_{p}$ -step-ahead reliance predictions ( $N_p \in \{15, 30, 45, 60\}$ $N_{p} \in {15, 30, 45, 60}$ time steps). The set-valued estimator consistently outperformed both an open-loop baseline and a particle filter (using Gaussian assumptions) in terms of Area Under the Receiver Operating Characteristic Curve (AUC-ROC).
- The performance gap widened at longer horizons (e.g., at $N_p=60$ , the set-valued estimator achieved a mean AUC of 0.82 vs. 0.76 for the particle filter).
- The authors attribute the particle filter's degradation to the compounding of errors from ad hoc Gaussian noise assumptions, whereas the set-based approach leverages the history of reliance observations through recursive intersection without distributional assumptions.
Computation: The average set-computation time was between 0.033 and 0.067 seconds, well below the 1-second sampling interval, confirming the potential for real-time online implementation.

Significance and Claims
The paper claims that this framework enables automation systems to be "continuously aware of, and responsive to, the human driver's state" without requiring the extensive data collection or distributional assumptions typical of probabilistic methods. By treating uncertainty as bounded rather than probabilistic, the approach is better suited for the sparse, asynchronous, and quantized nature of real-world human-automation data. The authors position this work as a necessary precursor to developing truly adaptive, human-aware automated vehicles that can intervene based on personalized cognitive states (e.g., adjusting take-over lead times based on trust levels). The paper concludes modestly, noting that future work is required to deploy this in closed-loop control policies and to incorporate additional measurements for full-state estimation.

The Paper's Story: A Detective with a Shrinking Box

Technical Summary: A General Set-Based Framework for Cognitive State Estimation

More like this