SketchVLM: Vision language models can annotate images to explain thoughts and guide users

Technical Summary: SketchVLM

SketchVLM is a training-free, model-agnostic framework designed to enhance the explainability and utility of Vision-Language Models (VLMs). It enables frontier models (like GPT-5 or Gemini-3-Pro) to produce non-destructive, editable SVG (Scalable Vector Graphics) overlays on input images to visually explain their reasoning and guide users through complex tasks.

1. The Problem: The "Black Box" of Textual Reasoning

Modern VLMs primarily communicate through text blocks. While highly capable, text-only responses present several challenges:

• Verifiability: It is difficult for users to quickly verify if a model's textual description (e.g., "the yellow handle near the center") accurately corresponds to the correct visual region.
• Instructional Clarity: For spatial or procedural tasks (e.g., "how to check car oil" or "maze navigation"), text is less intuitive than visual pointing, circling, or labeling.
• Limitations of Existing Solutions:
  • Image-editing models risk altering the source pixels, which can undermine trust and destroy the original context.
  • Fine-tuned sketching models often lack generalizability, failing when moved outside their specific training domains.
  • Coordinate-only models provide points but lack the expressive power of free-form drawing or labeling.

2. Methodology: The SketchVLM Framework

SketchVLM does not require retraining. Instead, it wraps existing frontier VLMs in a structured prompting and rendering pipeline consisting of three core components:

• Visual Prompting: To improve spatial precision, the framework appends a coordinate grid to the edges of the input image. This provides the model with a reliable reference system for $x$-$y$ positioning.
• Structured System Prompting: The model is instructed via a sophisticated system prompt to output its "thoughts" as a sequence of structured XML-style tags (e.g., <s1>, <s2>). These tags contain:
  • Primitives: Support for straight lines, Bézier curves (for smooth free-form drawing), rectangles, arrows, and text labels.
  • Parameters: Specific coordinates, timestamps ($t$-values) for curve fitting, font sizes, and colors.
• XML-to-SVG Conversion: A post-processing layer parses the model's XML output and converts it into a standardized SVG overlay. This layer uses a least-squares solution to fit cubic Bézier curves to the model's points, ensuring smooth, professional-looking annotations that are rendered as a separate layer over the original image.

The framework supports both single-turn generation (all annotations and the answer in one go) and multi-turn generation (iterative, step-by-step drawing to simulate human-AI collaboration).

3. Key Contributions

• Training-Free & Model-Agnostic: It leverages the inherent reasoning capabilities of frontier VLMs without the need for expensive fine-tuning.
• Non-Destructive Annotation: By using SVG overlays, the original image remains untouched, preserving data integrity and user trust.
• High-Fidelity Visual Reasoning: It introduces a way for models to perform "Visual Chain-of-Thought," where the sketch serves as a trace of the model's internal logic.
• Multi-Modal Versatility: The framework handles diverse tasks ranging from physics prediction (ball drop) and spatial reasoning (mazes) to object counting and part labeling.

4. Results and Evaluation

The authors evaluated SketchVLM across seven benchmarks (including maze navigation, ball-drop trajectory, and object counting) against state-of-the-art baselines like Nano Banana Pro (image-editing) and fine-tuned models (ViLaSR, ThinkMorph).

• Superior Accuracy: SketchVLMs improved visual reasoning task accuracy by up to +28.5 percentage points compared to alternative sketching approaches.
• Enhanced Annotation Quality: SketchVLM achieved up to 1.48× higher annotation quality and significantly higher scores in "Annotation–Text Alignment" (how well the drawing matches the text answer).
• Generalization: Unlike fine-tuned models that failed on unseen physics tasks (e.g., Ball Drop), SketchVLM maintained high performance due to the general reasoning power of the underlying frontier models.
• Efficiency: Single-turn generation was found to be nearly as accurate as multi-turn but significantly faster, making it practical for real-time applications.

5. Significance

SketchVLM represents a shift from text-centric AI to visually-grounded AI. By allowing models to "point and draw," it bridges the gap between abstract reasoning and human-like visual communication. This has profound implications for:

• Explainable AI (XAI): Making AI decisions transparent and verifiable.
• Human-AI Collaboration: Enabling interactive, step-by-step guidance in complex workflows (e.g., software tutorials, mechanical repairs, or educational tools).
• User Trust: Reducing the cognitive load required to verify AI outputs by providing intuitive, visual evidence.