The here-and-now of "real-time visual assistance". Assembling gestalt contexture in blind-sighted navigation

Technical Summary: The Here-and-Now of "Real-Time Visual Assistance"

Problem Statement
The paper addresses a critical gap in the design and understanding of "real-time visual assistance" for blind and low-vision individuals. While recent research in human-computer interaction and multimodal AI focuses on generating scene descriptions, there is a lack of empirical specification regarding what constitutes "real-time" in a practical, situated sense. Current systems often struggle with the "problem of relevance" (Dreyfus, 2007) or the "frame problem" (McCarthy & Hayes, 1969), failing to determine which environmental features are practically significant for an agent in motion.

The core issue is that existing approaches often treat scene descriptions as propositional reports of a pre-existing, static world. However, in dynamic, high-urgency settings like street crossings, the "here-and-now" is not a fixed scene to be described but a practical field of action that must be co-constructed. The paper investigates why artificial agents often fail to provide adequate assistance in these contexts compared to human guides, specifically focusing on how "real-time" is locally accomplished rather than merely measured by latency.

Methodology
The study employs a process-philosophical version of ethnomethodological conversation analysis (EMCA). The authors analyze a corpus of interactions involving blind participants navigating urban environments with two distinct types of assistance:

1. Artificial Agent: A blind participant using Meta Glasses (with Meta AI) to navigate a busy crossroad in Copenhagen.
2. Human Guide: A blind participant (a horse rider) navigating a crossroad with an experienced sighted instructor.

The data consists of video recordings and audio transcripts, analyzed using Jeffersonian conventions for talk and Mondadian conventions for embodied conduct. The analysis focuses on the sequential unfolding of interaction, examining how participants (both human and artificial) segment observable properties of the environment and how these segments are treated as relevant or irrelevant by the blind participant. The study uses the artificial agent's "discordant" conduct as a "Heideggerian" example (Garfinkel, 1996) to render visible the transparent work of achieving coherence in human-human interaction.

Key Findings and Analysis

• The Failure of "Detached" Description (Excerpt 1):
  In the interaction with the Meta Glasses, the artificial agent (MET) provided a description of the road as "clear" and free of obstructions. However, this description was produced with a temporal delay relative to the rapidly shifting traffic. Crucially, the agent's description was "detached" (Dreyfus, 1990); it treated the traffic as an object of observation rather than a dynamic constraint on the user's immediate action.

  • The blind participant (LAU) rejected the description not because it was factually incorrect regarding the moment the photo was taken, but because it failed to index the "vivid present" (Au-Yeung & Fitzgerald, 2023) of the crossing.
  • The agent's contribution was not "procedurally consequential" (Schegloff, 1992) for the ongoing activity. It failed to integrate the temporal and normative orders of the traffic (e.g., the imminent arrival of a car) into the user's "here-and-now."

• The Success of "Embedded" Guidance (Excerpt 2):
  In contrast, the sighted human guide (TRI) did not merely report features of the environment. Instead, she methodically segmented the surroundings into "front" and "behind" relative to the group's projected trajectory.

  • Her descriptions (e.g., "they are waiting") were not just factual reports but normative formulations that indexed the moral and temporal order of the situation.
  • The guide's contributions were embedded within multiple local orders: the conversational flow, the physical coordination of the horse riders, and the traffic flow.
  • By suspending conversation and using silence, the group created optimal conditions for hearing traffic, demonstrating that the "here-and-now" is a shared, constructed space-time rather than a geometric or clock-based reality.

• Gestalt Contexture as a Process:
  The study demonstrates that "gestalt contexture" in these settings is not a pre-existing visual whole but a processually assembled phenomenon. The sighted guide does not start with an inventory of discrete properties and assemble them; rather, they act within an organized phenomenal field where details gain sense through their place in the ongoing activity. The guide's grammatically declarative statements do not merely describe a scene; they progressively make selected features into constituent details of a shared, actionable reality.

Key Contributions

1. Re-specification of "Real-Time": The paper argues that "real-time" in visual assistance is not a technical metric of latency but a situated practice. A description is "real-time" only if it is timely relative to the practical urgency of the user's course of action and indexes the relevant temporal and normative orders of the environment.
2. Gestalt Contexture in Asymmetric Access: The study extends the concept of gestalt contexture from stable, symmetrically accessible objects to dynamic settings with asymmetric perceptual access (blind vs. sighted). It shows how coherence is achieved through the linguistic and embodied practices of the guide, rather than being immediately available to the perceiver.
3. Critique of AI Design Assumptions: The analysis reveals that current multimodal AI systems often fail because they treat the environment as a static object to be described (a "mirror floating with no foothold") rather than as a dynamic field of action. The paper highlights that responsiveness to human speech alone is insufficient; systems must orient to the "structures providing for the organization of the endogenous activity systems" (Goodwin, 1996).

Significance and Claims
The paper modestly claims to specify the "situated practices glossed by the terms 'real-time descriptions' and 'real-time visual assistance'." It does not propose a new algorithm or a specific technical architecture for future AI. Instead, it offers a praxeological critique of current design paradigms.

The authors argue that the dualistic distinction between "foregrounded" action and a passive "background" environment is empirically unsustainable. Environments, technologies, and bodies participate in the ongoing accomplishment of situations. Therefore, for assistive technologies to be effective, they must move beyond generating accurate, detached descriptions and toward participating in the co-construction of a shared "here-and-now" that is sensitive to the practical urgency and normative constraints of the user's activity. The study concludes that the "whole" of a traffic environment is progressively produced in and as the linguistic practices of the guide, a complexity that current AI systems struggle to replicate.