ASCII Art Turns LLMs into VLA Controllers

Technical Summary: ASCII Art Turns LLMs into VLA Controllers

Problem and Motivation
Vision-Language-Action (VLA) controllers typically extend Vision-Language Models (VLMs) with action supervision, requiring large-scale multimodal backbones, extensive data, and significant computational resources. While modular pipelines remain effective, VLA approaches aim to unify perception, instruction understanding, planning, and control into a single model. However, training and deploying VLM-based controllers is compute-intensive, and collecting high-quality robot demonstration data often relies on costly human teleoperation or suffers from sim-to-real discrepancies. This work addresses the question: Can a VLA model be built using only text-only Large Language Models (LLMs) without direct image inputs or multimodal training pipelines?

Methodology
The authors propose an "ASCII-as-Vision" interface that renders visual observations into text, enabling text-only LLMs to function as VLA controllers.

• ASCII-as-Vision Interface: RGB camera frames are converted into fixed-size, colored ASCII rasters (e.g., 96×54 resolution). This representation preserves task-relevant visual cues (objects, targets, spatial layout) and color information while remaining compatible with text-only LLMs. Background pixels are mapped to whitespace to reduce clutter.
• Observation Pipeline:
  • Simulation: Environments are procedurally generated using pygame. Scenes include movable objects, targets, a gripper, and static obstacles. Domain randomization is applied to colors, sizes, and rotations.
  • Real-World: A fixed overhead RGB camera captures the workspace. A deterministic pre-processing pipeline includes camera calibration, workspace cropping (bounded by AprilTags), color-based masking, denoising, and a parallax correction step to remap the gripper center to the tabletop reference frame before ASCII encoding.
• Action Space: The policy outputs a single command in a constrained Domain-Specific Language (DSL):
  • MOVE(x, y): Discrete grid coordinates.
  • GRIP(PICK) / GRIP(DROP): Grasping actions.
  • UNREACHABLE(BLOCKED|NARROW|UNDEFINED): Explicit declarations of task infeasibility (blocked paths, narrow gaps, or missing entities).
• Learning Pipeline:
  • Stage 0 (Bootstrap): Supervised fine-tuning using demonstrations from an expert teacher policy based on traditional motion planning (A*).
  • Stage 1–20 (Iterative Improvement): The DAgger (Dataset Aggregation) algorithm is employed. The current policy interacts with the environment, and the teacher provides corrective labels for visited states to address compounding errors.
  • Fine-tuning: Models are adapted using Low-Rank Adaptation (LoRA).

Key Contributions

1. Novel ASCII-based VLA: The authors propose and evaluate a VLA controller that conditions on visual state via ASCII text, eliminating the need for native image inputs or multimodal backbones. The controller also evaluates task reachability.
2. Reproducible Pipeline: A complete training framework is provided, including an environment generator, a motion planning teacher, and a DAgger-style iterative data collection loop.
3. Comprehensive Benchmarking: The study benchmarks multiple text-only LLMs and VLMs across various families and scales in simulation and demonstrates successful sim-to-real transfer on a physical 7-DOF manipulator.

Results
Experiments were conducted in a 2D top-down manipulation environment with five reachability regimes (Reachable-Simple, Reachable-Complex, Unreachable-Blocked, Unreachable-Narrow, Unreachable-Undefined).

• Performance Gains: Stage-wise aggregation (DAgger) yielded substantial improvements over the Stage 0 bootstrap. For instance, Qwen3-4B improved from 30.8% to 89.2% success on test episodes.
• Model Comparison:
  • The strongest overall performance was achieved by Qwen3-VL-4B-Instruct (83.6% success, 8.6% collision rate).
  • Among text-only models, Qwen3-14B achieved 81.2% reachable success.
  • Crucially, at comparable scales, text-only LLMs were competitive with VLMs. In some cases, text-only models outperformed their VLM counterparts (e.g., Qwen3-8B slightly outperformed Qwen3-VL-8B-Instruct in reachable success and false-positive unreachability rates).
• Unreachability Detection: Models achieved high detection rates (mid-to-high 90%) and subtype accuracy for infeasible tasks.
• Real-World Deployment: The learned controllers were successfully deployed on a Flexiv Rizon4s robot arm. The system handled pick-and-place tasks with minimal changes to the policy interface, relying on the deterministic pre-processing pipeline to align real-world observations with the simulator's ASCII representation.

Significance and Claims
The paper claims that ASCII rendering serves as a lightweight, interpretable modality bridge from images to text. It demonstrates that text-only LLMs can effectively replace multimodal backbones for 2D manipulation tasks, potentially lowering the engineering barriers and computational costs associated with VLA research. The authors note that while ASCII art discards some visual nuance, it allows for the utilization of widely available LLMs and overcomes data gathering constraints. In the specific 2D manipulation context studied, text-only models even performed better than equivalent visual models.

Limitations
The authors acknowledge that ASCII rendering is an intentional information bottleneck, potentially limiting performance in cluttered scenes, small-object interactions, or tasks requiring precise geometry. They also note that optimization results are model-dependent (e.g., Gemma models showed instability compared to Qwen3/Llama-3.2) and that geometric clearance reasoning (e.g., in "Unreachable-Narrow" scenarios) remains a challenge. Future work is suggested for higher-resolution encodings, hybridization with classical planners, and extending the interface to RGBD inputs.