Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images

Technical Summary: Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images

Problem Statement
Long source-code contexts often exceed the token limits of language models, motivating proposals to render code as images for vision-language models (VLMs). While recent research investigates whether models can solve code tasks after this transformation, a critical systems question remains unanswered: how do commercial API providers count the resulting requests? Specifically, it is unclear how input-token accounting changes when code is transmitted as raw text versus compact rendered images, how this relationship scales with source length, and whether "visual compression" offers a net reduction in reported tokens across different providers and model aliases.

Methodology
The study employs a reproducible, black-box measurement protocol to compare input-token accounting across three major providers: Anthropic, OpenAI, and Google Vertex AI.

• Corpus: The dataset consists of five revision-pinned source files (Python, JavaScript, Rust, Go, and Java) from prominent open-source projects. These files are sliced into nine nested prefixes ranging from 20 to 2,000 lines.
• Treatment (Compact Image): The image arm applies a two-stage transformation:
  1. Indentation Compression: Leading spaces are replaced with compact markers (e.g., > for 4-space indents, ^N for irregular runs).
  2. Rendering: The transformed text is rendered as PNG pages.
• Experimental Design: For each source size and language, paired requests are sent to 15 available model aliases (4 Anthropic, 6 OpenAI, 5 Gemini). Both arms include an identical one-sentence summarization instruction ("Summarize what this code does in one sentence").
• Metrics: The primary metric is the ratio of provider-reported image input tokens ($I$) to text input tokens ($T$). The study reports weighted aggregate ratios (summing all tokens across the dataset) and size-stratified ratios to identify break-even points.
• Constraints: The study explicitly isolates token accounting from semantic fidelity, task accuracy, latency, monetary cost, or coding-agent efficiency. It does not claim that image tokens are computationally equivalent to text tokens.

Key Contributions

1. Reproducible Artifact: A dataset of 1,350 successful API calls and 675 complete text/image pairs, including raw usage records, validators, and deterministic analysis scripts.
2. Size-Stratified Measurements: Empirical evidence showing that token reduction is not uniform; it varies significantly by source length and provider.
3. Modality Audit: A targeted investigation revealing non-monotonic behavior in image accounting, specifically at page boundaries.
4. Validity Boundary: A clear demarcation between token counting metrics and information preservation, preventing the conflation of "fewer tokens" with "better performance" or "lower cost."

Results

• Aggregate Reductions: Across the full benchmark, compact images receive significantly fewer reported input tokens than raw text:
  • Anthropic: 0.135 ratio (86.5% reduction).
  • OpenAI: 0.194 ratio (80.6% reduction).
  • Gemini: 0.242 ratio (75.8% reduction).
• Break-Even Behavior: The aggregate ratios conceal critical differences in scaling:
  • Anthropic and OpenAI: Image inputs receive lower token counts than text at every tested size (20 to 2,000 lines).
  • Gemini: Images incur a massive overhead for short contexts. At 20 lines, Gemini images require 6.95 times more tokens than text. The image approach only becomes advantageous (crossing below parity) at 200 lines.
• Model Aliases: Within providers, many model aliases share identical accounting signatures (e.g., all six OpenAI models in the study returned identical aggregate ratios), suggesting shared internal accounting rules rather than independent model behaviors.
• Non-Monotonicity: A targeted audit of Gemini revealed that reported image token counts can change non-monotonically at page boundaries. For example, increasing the source from 800 to 1,200 lines (adding a second page) resulted in a decrease in reported image tokens for one model, contradicting the expectation that token counts scale linearly with content.

Significance and Claims
The paper claims that while compact image rendering can drastically reduce reported input tokens for long contexts, the benefit is highly conditional:

1. Provider Specificity: There is no universal "visual compression" advantage. Providers like Gemini exhibit high fixed costs that make images counterproductive for short contexts.
2. Routing Implications: A coding harness cannot rely on a global policy to "send code as images." Instead, routing logic must be calibrated per provider and source size, potentially falling back to text for short contexts or when page boundaries introduce discontinuities.
3. Limitations of Token Counts: The study emphasizes that a reduction in reported tokens does not imply equivalent information content, lower monetary cost, or reduced compute. Indentation markers may be misdecoded, and rasterization may obscure punctuation or structure.
4. Future Work: The authors position this study as a "measurement surface" upon which future research can build. They argue that the next necessary step is to cross representation with task quality (e.g., exact transcription, defect localization) to determine if the token savings translate to actual coding utility.

The paper concludes that compact rendered code is a viable strategy for token accounting in specific regimes (long contexts, specific providers) but requires careful, provider-specific calibration and further validation regarding information fidelity.