Toward Governing Perception in Safety-Critical Mediated Reality on the Move

This position paper argues that as wearable Augmented Reality evolves into Mediated Reality capable of altering visual scenes in safety-critical mobile contexts, it is essential to develop governance mechanisms that allow users to configure, inspect, and understand these perceptual modifications without compromising safety and trust.

The Gist

Imagine you are driving a car, but instead of looking through a normal windshield, you are looking through a smart, magical window (like high-tech glasses).

Currently, most of these smart windows work like a highlighter pen. If a pedestrian steps out, the window draws a bright box around them. If a car is turning, it draws an arrow. This is called Augmented Reality (AR). It adds information on top of the real world.

But this paper argues that we are moving into a new era called Mediated Reality (MR). Instead of just adding highlights, this new technology can erase, change, or swap out parts of the world you see.

Think of it like a Photoshop editor for your real-life vision:

• Diminished Reality: The window could magically delete a distracting billboard or a parked car blocking your view, making the road look cleaner.
• Modified Reality: It could turn a confusing, messy intersection into a simple, cartoon-like diagram so you understand it instantly.

The Problem: Who is the Editor?

The author, Pascal Jansen, points out a big danger: If the window changes what you see, you might miss something important.

Imagine you are driving, and the window decides to "clean up" the view by removing all the parked cars on the side. You feel safe and relaxed. But suddenly, a child runs out from behind one of those "deleted" cars. Because the window erased the car, you didn't see the child. You trusted the window too much.

This paper asks: How do we make sure the user stays in charge of this magical window? We need a way to "govern" (control and understand) what the window is doing, without getting distracted while driving.

The Three Big Challenges

The paper suggests three main things we need to figure out to make this safe:

1. When can you touch the controls? (The "Hands-Free" Rule)
You can't be fiddling with menus while driving at 60 mph.

• The Analogy: Think of it like a flight mode on a phone. You set your preferences before you take off (e.g., "I want a clean view" or "Show me everything").
• During the trip: You can only make tiny, quick adjustments (like a voice command saying "Show me the full scene" or looking at a hidden object to reveal it).
• After the trip: You get a "replay" of the drive that shows you exactly what the window changed, so you can learn from it.

2. How much control should you have? (The "Remote Control" vs. "Dial" Dilemma)
Should you have a simple button that says "Simplify View" or a complex dial where you can adjust exactly which colors are changed?

• The Analogy: If you give a non-expert a complex mixing board with 50 sliders, they might accidentally turn off the "pedestrian alarm" while trying to fix the "music volume."
• The Solution: We need a "Smart Assistant" approach. The system suggests changes ("Hey, it's foggy, should I simplify the view?"), and you just say "Yes" or "No." This keeps it simple but still lets you have a say.

3. How do you know what's real? (The "Truth Label")
If the window changes the world, how do you know what is real and what is fake?

• The Analogy: Imagine watching a movie where the actors are real, but the background is a green screen. You need a subtle signal (like a tiny icon) that says, "The background here is computer-generated."
• The Challenge: If the signal is too loud, it distracts you. If it's too quiet, you don't notice it. We need a way to signal, "I removed this trash can because it was distracting," without cluttering your vision.

The Big Picture: Who is Responsible?

The paper ends with a serious question about blame.
If an accident happens, and the smart window had erased the object that caused the crash, who is responsible? The driver? The car company? The software?

To fix this, the system needs to keep a black box log. It must record exactly what it changed, why it changed it, and what the driver saw at that moment. This way, if something goes wrong, we can look back and say, "Ah, the system removed that obstacle, and that's why the driver didn't react."

Summary

This paper is a call to action. It says: Don't just let the technology change your reality in secret. We need to build "smart windows" that:

1. Let you set the rules before you start.
2. Show you clearly what they are hiding or changing.
3. Keep a record of their actions so we know who is responsible.

The goal is to make our future travel safer, not just by adding more information, but by carefully curating what we see, while keeping us in the driver's seat of our own perception.

Technical Summary

Based on the position paper "Toward Governing Perception in Safety-Critical Mediated Reality on the Move" by Pascal Jansen, here is a detailed technical summary.

1. Problem Statement

The paper addresses the evolving landscape of wearable technology in dynamic, safety-critical mobility contexts (e.g., automated driving, cycling, pedestrian navigation).

• Limitation of Current AR: Existing Augmented Reality (AR) systems primarily rely on additive overlays (highlighting hazards or intentions). While useful, they add visual information to an already complex environment, potentially increasing cognitive load and clutter.
• The Shift to Mediated Reality (MR): Advances in Head-Mounted Displays (HMDs) and computer vision (object detection, inpainting) now enable Diminished Reality (DR) (suppressing/removing elements) and Modified Reality (ModR) (transforming/substituting elements).
• The Core Challenge: Unlike additive AR, MR fundamentally alters the perceptual availability of the environment. By removing or changing scene elements, MR reshapes the evidential basis for Situation Awareness (SA) and Trust Calibration.
  • If a user cannot see an object because it was suppressed, they cannot form SA Level 1 (perception) regarding that object.
  • If predictions are visualized without distinction from reality, users may misinterpret probabilistic forecasts as certain facts.
  • The Gap: There is currently no framework for users to govern (configure, inspect, and understand) these perceptual mediations without compromising safety or attention. Users risk "epistemic opacity," where they do not know what the system is hiding or altering, leading to misaligned trust and safety risks.

2. Methodology

This paper is a Position Paper presented at a workshop. It does not report on a specific empirical experiment or user study with quantitative results. Instead, it employs a conceptual analysis and theoretical framing approach:

• Literature Synthesis: It integrates concepts from Human-Computer Interaction (HCI), Mediated Reality (Mann's definition), Situation Awareness (Endsley's levels), and Trust Calibration in automation.
• Contextual Analysis: It analyzes the specific constraints of "on-the-move" environments (time pressure, limited attention, mechanical constraints) compared to stationary AR contexts.
• Problem Decomposition: The author breaks down the governance challenge into four distinct design dimensions:
  1. Timing of control (When/How).
  2. Granularity of control (What).
  3. Epistemic signaling (How to communicate uncertainty/status).
  4. Accountability (Responsibility attribution).

3. Key Contributions

The paper proposes a research agenda and a design framework for Governable Mediated Reality. Its primary contributions are:

A. Conceptual Shift: From "What to Add" to "What is Available"

The paper argues that the design space must shift from simply adding information (AR) to managing the perceptual availability of the scene (MR). This requires treating scene suppression and transformation not as hidden rendering choices, but as interaction-level variables that users must be able to inspect and control.

B. The "Three Loci of Control" Interaction Model

To address the impossibility of continuous in-situ governance during high-speed motion, the paper proposes a temporal separation of control:

1. Pre-ride Configuration: Users set high-level mediation profiles (e.g., "reduced clutter," "conservative uncertainty") via low-demand interfaces (voice, dashboard) before entering motion.
2. Minimal In-situ Override: During motion, only lightweight, low-cognitive-load interventions are allowed (e.g., a single voice command to "show full scene," gaze-triggered reveal of a suppressed object, or context-anchored affordances).
3. Post-ride Inspection: A structured summary and event-based replay mechanism allow users to review when and why transformations occurred, binding interventions to spatiotemporal anchors (timestamp, location, maneuver).

C. Design Challenges Framework

The paper outlines four critical research challenges:

• Governance Timing: Determining safe moments to surface controls (e.g., low traffic density) and designing replay mechanisms that help users connect post-hoc explanations to their prior attentional state.
• Granularity of Control: Balancing coarse control (e.g., "simplified environment" presets) which is easy to understand but epistemically opaque, against fine-grained control which offers agency but risks complexity. The paper suggests mixed-initiative approaches where the system proposes adjustments based on context (e.g., high density) for user confirmation.
• Epistemic Signaling: Developing visual encodings that communicate the status of elements (sensed vs. predicted vs. filtered) without causing visual noise or inattentional blindness. Techniques like gaze-triggered inspection and transient overlays are proposed.
• Responsibility & Accountability: Establishing logging systems that record suppression/transformation events, triggers, and confidence levels. This is crucial for post-incident analysis to determine if a user had sufficient perceptual access to react to a hazard.

4. Results

As a position paper, there are no empirical results (e.g., user performance metrics, trust scores). However, the paper presents theoretical findings and proposed research questions:

• Finding: DR and ModR are not merely technical extensions but interaction variables that fundamentally alter how users construct Situation Awareness and appraise system capability.
• Finding: Current AR design principles (additive overlays) are insufficient for MR; new paradigms for "governability" are required to prevent non-calibrated trust.
• Proposed Research Questions:
  • How does control granularity affect SA and trust under time pressure?
  • What UI mechanisms (e.g., delta visualizations) make "what changed" legible enough to prevent epistemic opacity?
  • How can epistemic signaling remain behaviorally effective without degrading scene clarity?
  • What logging abstractions provide actionable accountability for mediated incidents?

5. Significance

This paper is significant for the future of Mixed Reality (MR) and Automated Mobility:

• Safety Criticality: It highlights that in safety-critical domains, the absence of visual information (via DR) is as dangerous as false information. Governance mechanisms are essential to ensure users do not develop a false sense of security or miss critical hazards.
• Trust Calibration: It provides a pathway to align user reliance with system capability. By making mediation transparent, users can better understand the system's limitations and the nature of the environment.
• Research Agenda: It reframes MR in mobility not just as a display technology, but as a perceptual mediation system that requires rigorous governance, auditability, and intelligibility. This sets the stage for future HCI research into "governable perception" in dynamic environments, influencing the design of next-generation HMDs for driving, cycling, and navigation.