Technical Summary: Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents

Problem Statement

Modular visual reasoning systems increasingly rely on shared working memory (e.g., "whiteboards" or scratchpads) to externalize intermediate states, allowing limited-capacity models to decompose complex queries into verifiable steps. However, the efficacy of this paradigm remains unproven under resource-constrained regimes, specifically when utilizing weak learners (4B–8B parameter models) with strict compute budgets.

The central hypothesis of this work is that in such low-capacity settings, the collaborative loop acts not as a reliable memory store, but as a noisy communication channel. Every additional interaction turn incurs linear computational costs while introducing risks of noise accumulation, where ungrounded intermediate notes are "hardened" into incorrect evidence, or policy drift, where the model shifts toward under-specified outputs. The paper argues that without rigorous integrity auditing, the failure dynamics of intermediate state evolution in these regimes remain underexplored.

Methodology: The CoSee Framework

To investigate these limits, the authors introduce CoSee, an auditing framework and collaboration protocol designed to measure the utility of shared context under small-model constraints.

1. Formalization of the Inference Loop

The problem is formulated as a sequential decision-making process where a vision-language model $M_\theta$ generates an answer $a$ conditioned on visual input $I$ and query $q$, mediated by a sequence of discrete latent states (the Board, $B_{1:T}$). The framework treats the inference trajectory as a probabilistic trace $\tau = (I, q, B_{1:T}, \hat{a})$.

2. Experimental Protocols

The study compares three distinct inference protocols under strictly matched prompts and decoding caps (max 32 new tokens per call, $T=3$ steps):

• Direct Inference (Baseline): A single-turn generation without intermediate state tracking.
• Open-Loop Iterative Refinement:
  • Single-Agent: One agent iteratively updates the board.
  • Dual-Agent (Scanner–Checker): Two roles derived from the same backbone operate in alternating turns (one proposes evidence, the other refines/refutes).
• Gated Transition (Verified-Board): A conditional gating mechanism where a verification function $V_\phi$ (parameterized by the same backbone) filters proposed board entries. Only entries marked as "Supported" are appended to the board; others are discarded. This acts as an information bottleneck to prevent noise propagation.

3. Evaluation Setup

• Backbones: Experiments utilize 4B–8B class models: Qwen3-VL-4B/8B-Instruct, Phi-4-multimodal-instruct, and Gemma-3-4B-it.
• Benchmarks: Three distributions representing distinct failure modes:
  • SlideVQA: Multi-page retrieval (high-recall regime).
  • ChartQAPro: Quantitative reasoning with adversarial perturbations (high-precision regime).
  • VQAonline: Open-ended synthesis from web data (policy-sensitive regime).
• Metrics: Beyond standard Exact Match (EM) and Loose Match, the authors employ Format-Robust Scoring (to isolate hallucination from formatting errors) and Policy-Aware F1 (to detect coverage collapse). Crucially, they enforce a "no-exclusion" scoring policy, retaining malformed or leaked outputs as failures to penalize protocol instability.
• Cost Analysis: Performance is evaluated against computational cost using Cost–Accuracy Pareto frontiers, plotting utility (accuracy) against total generated tokens.

Key Results

The empirical investigation reveals a distinct Efficiency Paradox: naive usage of shared memory often degrades performance rather than improving it for weak learners.

1. Negative Returns on Compute

Across all three backbones and datasets, the multi-agent and single-board open-loop protocols frequently underperform the direct single-turn baseline.

• ChartQAPro & VQAonline: Naive collaboration shows statistically significant performance drops. The additional compute (tokens) correlates negatively with accuracy, placing these methods in the "negative utility" quadrant of the Pareto frontier.
• SlideVQA: While retrieval tasks show marginal or neutral gains, they do not exhibit the severe degradation seen in reasoning-heavy tasks, suggesting the failure mode is specific to reasoning chains rather than simple evidence retrieval.

2. Identified Failure Modes

Trace-level diagnostics isolate two dominant mechanisms driving this degradation:

• Noise Reinforcement (ChartQAPro): In quantitative reasoning, ungrounded intermediate notes (e.g., misreading a chart legend) are written to the board and subsequently treated as ground-truth evidence by the final answerer. This creates a positive feedback loop where early perception errors are "hardened" into the final answer.
• Policy Collapse (VQAonline): In open-ended synthesis, the presence of intermediate context induces a distributional shift. The model collapses toward overly terse responses (shifting from 17–32 token bins to 0–16 token bins), leading to severe under-coverage of required answer components. This is a learned dependency on the board summary rather than a result of token truncation.

3. The Role of Verification

The Verified-Board control demonstrates that the bottleneck is not reasoning capacity but communication fidelity. Introducing a simple verification gate (Support/Refute check) recovers the majority of the performance drop on ChartQAPro, effectively arresting error propagation. This confirms that lightweight verification is the minimal necessary condition for reliable scaling in this regime.

Significance and Contributions

The paper makes three primary contributions to the understanding of modular visual agents:

1. Formal Auditing Framework (CoSee): It provides a rigorous protocol for tracing information flow and integrity in modular VQA systems, enabling systematic investigation into the limits of weak learner collaboration.
2. Taxonomy of Failure Mechanisms: It establishes a classification of dominant failure modes—specifically Noise Reinforcement and Policy Collapse—demonstrating that naive shared workspaces can amplify hallucinations rather than resolve them.
3. Design Principles for Resource-Constrained Agents: The findings challenge the intuition that "thinking on paper" inherently improves reasoning. Instead, the paper argues that for 4B–8B models, the bottleneck lies in communication fidelity, not reasoning depth. It posits that reliable modular design requires closed-loop controls (gated transitions) and grounded information bottlenecks to prevent the thermodynamics of error propagation, rather than simply proliferating open-loop roles.

In conclusion, the work suggests that for resource-constrained agents, increased test-time compute without explicit verification mechanisms yields diminishing or negative returns. The path toward reliable modular agents lies in engineering mechanisms that filter noise at the source, ensuring that shared state acts as a valid reasoning support rather than a noisy channel.