Technical Summary: VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following

Problem Statement

While Vision-Language Models (VLMs) demonstrate strong performance on broad multimodal benchmarks, they often lack robust control over basic visual operations. Specifically, this paper investigates line tracing: the ability of a model to follow a selected visual path through successive local continuations. The authors argue that current VLMs struggle to maintain the identity of a selected path when nearby competing alternatives are present, even in settings devoid of complex topological features like crossings or semantic ambiguities. This failure is distinct from general spatial reasoning deficits; it is a specific inability to sustain attention on a continuous structure against local visual distractors.

Methodology

The study employs a controlled diagnostic approach to isolate the tracing mechanism from confounding factors such as semantic shortcuts, long-range dependencies, and complex topology.

1. Controlled Tracing Tasks

The authors designed two primary tasks to strip away external difficulties:

• Modified Circuit Connection Task: Based on prior smooth visual search work, this task removes wire crossings and overlaps. Models must trace a specific wire from a starting port to a component, reporting a sequence of colored dots placed along the wire. This isolates the ability to follow a single path in a multi-wire environment.
• Swirl Task: An abstract dataset consisting of a single spiral path with eight uniquely colored dots. The task varies rotation density (Low, Moderate, High) to control the proximity of neighboring turns. This eliminates domain-specific priors (like circuit ports) and focuses purely on the difficulty of distinguishing the correct local continuation from a visually similar adjacent turn.

2. Diagnostic Interventions

To determine the mechanism of failure, the authors conducted several analyses:

• Local Similarity Stress Tests: In the Swirl task, they constructed conditions where nearby distractors varied in structural similarity to the target path (e.g., "Shared segment" vs. "Different all").
• Causal Masking: They applied occlusion masks to specific regions (the distractor, the target path, or random patches) to test if removing the competing signal restores performance.
• Internal Representation Analysis: Using open-source models (Qwen3-VL and InternVL3.5), they measured:
  • Attention Preference: Whether the model's attention from the current position favors the true next dot over a distractor.
  • Representational Preference: Whether the hidden state of the current position is geometrically closer to the true continuation or the distractor in the embedding space.
• Remedy Evaluation: The paper tested standard mitigation strategies:
  • Model Scaling: Evaluating models of varying sizes (2B to 235B parameters) within the Qwen3-VL family.
  • Reasoning: Testing reasoning-enabled models to see if step-by-step deliberation improves tracing.
  • Explicit Instructions: Providing prompts that explicitly forbid proximity-based selection and mandate line-following.

3. Generalization Tests

To verify that failures are not artifacts of synthetic data, the authors evaluated models on:

• HANDLOOM Dataset: Tracing tangled cables with self-overlaps and crossings.
• Metro Maps: Tracing routes on transit maps with dense labels and intersecting lines.

Key Results

1. Systematic Path-Switching Failures

Even state-of-the-art VLMs (including Claude Sonnet 4.5, GPT-5.4, and Gemini3-Flash) fail significantly on these tasks.

• Circuit Task: Claude Sonnet 4.5 achieved only 6.2% accuracy. The dominant error mode (90–98% of errors) was an adjacent-wire jump, where the model starts correctly but switches to a nearby wire.
• Swirl Task: Performance degrades sharply with rotation density. At high density, GPT-5.4 drops to ~20%, and Claude Sonnet 4.5 and Qwen3-VL-235B reach 0%. The dominant error is an adjacent-turn jump.

2. Mechanism: Local Competition

The failures are driven by local visual competition, not random noise or global reasoning limits.

• Similarity Sensitivity: Accuracy drops significantly only when the distractor shares local structure (e.g., a parallel segment) with the target path. When similarity is removed, performance recovers.
• Causal Evidence: Masking the distractor region increases accuracy by ~13.5 points, while masking the target path destroys performance. This confirms the distractor actively pulls the model off course.
• Internal State Drift: Internal analyses reveal that as local similarity increases, the model's attention and hidden representations shift away from the true continuation and toward the distractor. In "Shared segment" conditions, the internal state is often closer to the distractor than the true path.

3. Ineffectiveness of Standard Remedies

• Scaling: Increasing model size (up to 235B parameters) yields only marginal gains, which diminish under higher difficulty.
• Reasoning: Reasoning-enabled models do not perform genuine visual tracing. Instead, they rely on heuristic substitutes (e.g., "ring decomposition" or "angular matching") to guess the sequence. While this sometimes improves accuracy, it comes at a massive token cost (thousands of tokens for a simple trace) and does not eliminate adjacent-jump errors (which remain 71–84% of failures).
• Instructions: Explicitly instructing models to follow the line and ignore proximity decreases accuracy across all models, suggesting that current architectures cannot simply be prompted into reliable tracing.

4. Real-World Persistence

The same path-switching failure persists in complex, real-world settings like tangled cables and metro maps. Models frequently switch to nearby cables or stations, and reasoning traces often show models reconstructing routes based on global cues rather than local verification.

Significance and Claims

The paper claims that robust visual tracing under local competition is a fundamental, unresolved bottleneck in current VLM architectures.

• Diagnostic Value: The authors propose that line tracing serves as a concrete, measurable diagnostic for the ability to maintain connected visual structure.
• Architectural Limitation: The failure is not due to a lack of reasoning capacity or training data size, but rather a lack of dedicated mechanisms to preserve path identity against local distractors. Current models appear to rely on language priors or coarse statistics rather than genuine, step-by-step visual continuation.
• Practical Necessity: As VLMs are deployed in applications requiring reliable path following (e.g., reading transit maps, tracking deformable objects, analyzing flowcharts), addressing this specific failure mode is a practical necessity, not just a theoretical concern.

The authors conclude that future research must focus on developing mechanisms that can maintain the selected path across successive continuations, as current architectures fail to do so reliably even when the task is stripped of all other complexities.