🤖 AI

Who Leads Now? Token-Level Modality Arbitration for Chart-to-Code Generation

The paper introduces MoCA, a novel framework that employs a Cross-modal Arbitration Block to dynamically separate and coordinate distinct visual and code branches at the token level, thereby overcoming the interference inherent in existing chart-to-code methods and achieving superior performance through structured modality arbitration.

Original authors: Qinghao Fu, Yarong Wang, Shunlei Ning, Yilin Wang, Shunwen Bai, Xinda Wang, Jiaotuan Wang, Yinan Nie, Wei Zhou

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Qinghao Fu, Yarong Wang, Shunlei Ning, Yilin Wang, Shunwen Bai, Xinda Wang, Jiaotuan Wang, Yinan Nie, Wei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, data is often presented as charts: bar graphs, line plots, and pie charts that turn numbers into visual stories. For decades, computers have been excellent at reading these images to answer simple questions, like "what is the highest value?" or "what color is the largest slice?". However, a more difficult challenge has emerged: asking a computer to not just read a chart, but to rebuild it from scratch using code. This task requires a machine to look at a static image, understand the hidden data and the specific design choices behind it, and then write a set of instructions that can draw an identical picture on a screen. It is a test of two very different skills at once. The machine must be a sharp-eyed observer, capable of noticing subtle details like the exact shade of a line or the precise position of a label, and it must also be a disciplined architect, capable of writing logical, error-free code to reproduce those details.

For a long time, researchers tried to teach computers to do both things at once, blending visual understanding and coding ability into a single, massive brain. The assumption was that a model could learn to see and write code simultaneously. Yet, this approach often led to confusion. When a chart was visually complex, the model might struggle to write the code. When the chart required complex logic, the model might fail to see the details. The two skills, while necessary for the same job, seemed to get in each other's way when forced to share the same internal pathways. A new study from researchers at Zhejiang University, Ant Group, and other institutions suggests that the solution is not to blend these abilities, but to keep them separate and let them take turns leading the work.

The researchers, led by Qinghao Fu and Wei Zhou, developed a system they call MoCA. Instead of forcing a single model to do everything, they built a framework with two distinct pathways: one dedicated to visual understanding and another dedicated to coding. Imagine a construction crew where one team specializes in inspecting the raw materials and the other in the actual building. In MoCA, these two teams work side by side, but they do not merge into a single confused entity. Instead, a small, lightweight decision-maker, which the authors call an arbiter, watches the work in real time. With every single word of code the system generates, this arbiter decides how much help to take from the visual team and how much to rely on the coding team.

This decision-making happens constantly and changes from moment to moment. When the system is trying to figure out the color of a specific bar in a chart, the arbiter leans heavily on the visual pathway. When it needs to decide how to stack those bars or calculate the total height, the arbiter shifts its trust to the coding pathway. The researchers found that this dynamic switching is not random; it follows a structured pattern. In the early stages of generating a chart, the system relies more on visual processing to understand the input. As it moves into the middle of the task, it shifts toward code generation to build the structure. Finally, as it finishes, it may return to visual refinement to ensure the details match the original image. This fluid coordination allows the system to handle charts that are visually messy as well as those that are logically complex, without getting stuck.

To teach this system how to work, the researchers used a two-step training process. First, they showed the model thousands of examples of charts and their corresponding code, but they did not just ask for the final code. They also provided a "thought process," a step-by-step breakdown of how to look at the chart and translate it into instructions. This helped the model learn to connect what it saw with what it needed to write. In the second stage, the system was allowed to practice on its own. It generated code, and if the code ran successfully and the resulting image looked like the original, it received a reward. If the code failed or the image was wrong, it was corrected. This cycle of trial and error, guided by specific rewards for both the logic of the code and the visual accuracy of the result, refined the system's ability to coordinate its two branches.

The results of this approach were tested against several other advanced models on three different benchmarks designed to evaluate chart-to-code generation. The new system, MoCA, outperformed the competition in key areas. On one major test, it successfully generated working code for 88.83 percent of the charts, a significant improvement over other specialized models. It also scored higher on metrics that measured how well the generated code matched the text and structure of the original chart. Perhaps most importantly, the researchers showed that these gains did not come simply because the model was larger or had more training data. When they tested versions of the system that used the same amount of computing power but lacked the dynamic switching mechanism, the performance dropped. This confirmed that the secret to success was not just having more brainpower, but having a smarter way to organize it.

The study also revealed that the system's decision-maker learned to be very specific. It did not apply a fixed rule, such as "always use 50 percent visual help and 50 percent code help." Instead, it adjusted its strategy based on the specific chart it was looking at and the specific part of the code it was writing. For some charts, the visual branch might dominate the entire process. For others, the code branch might take the lead almost immediately. This flexibility allowed the system to adapt to the unique demands of each task, whether it was a simple line graph or a complex, multi-layered diagram.

By separating the skills of seeing and writing, and then giving them a dedicated manager to coordinate their efforts, the researchers solved a problem that had stumped previous attempts. They demonstrated that for complex tasks like rebuilding a chart from an image, the whole is greater than the sum of its parts only if the parts are allowed to function in their own strengths. The system does not try to be a master of everything at once; instead, it knows exactly when to look and when to write, producing results that are both visually faithful and logically sound. This approach offers a new blueprint for how artificial intelligence might handle other tasks that require a mix of very different kinds of intelligence, moving beyond the idea of a single, all-purpose model toward a more coordinated and adaptable future.

Technical Summary: MoCA – Token-Level Modality Arbitration for Chart-to-Code Generation

Problem Definition

Chart-to-code generation requires a model to interpret fine-grained visual details of a chart image and produce executable plotting code that faithfully reproduces the original. This task presents a unique challenge: the requirements for visual grounding (recognizing labels, colors, layouts) and code generation (constructing logic, axes, and structures) are not uniform across all inputs.

Visually Complex Charts: Require precise recognition of dense labels and subtle visual variations but may have simpler plotting logic.
Code-Demanding Charts: May have simple visuals but require complex logical structures (e.g., stacked bars, custom axes).

Existing approaches typically either train visual and coding abilities separately or fine-tune a single model where these abilities are entangled. Neither strategy explicitly models the dynamic interaction between these distinct capabilities, often leading to failures where a model misreads visual details or generates syntactically incorrect code. Furthermore, existing model-merging techniques often rely on static, input-independent parameter coefficients, failing to adapt to the varying demands of specific chart instances.

Methodology: MoCA (Mixture of Cross-modal Arbitration)

The authors propose MoCA, a framework designed to separate visual and code capabilities into distinct pathways while dynamically coordinating their contributions during generation.

1. Architecture: Cross-modal Arbitration Block (CAB)

MoCA is built upon a dual-branch architecture where a Visual Branch and a Code Branch operate as separate transformations within the model.

Initialization: The framework uses Qwen2.5-VL-7B as the multimodal backbone and Qwen2.5-Coder-7B to initialize the code branch. Specifically, the Visual Branch is initialized from the Feed-Forward Network (FFN) parameters of the vision-language model, while the Code Branch is initialized from the corresponding FFN parameters of the code model.
The Arbiter: At the core of MoCA is a lightweight Arbiter within each CAB. For every token $t$ at every layer $l$ , the Arbiter takes the intermediate hidden representation $h_{i,l,t}$ and predicts branch weights $\alpha_{i,l,t} = (\alpha^{vis}, \alpha^{code})$ .
Dynamic Coordination: The output of the CAB is a weighted sum of the two branches:
$z_{i,l,t} = \alpha^{vis}_{i,l,t} B^{vis}(h_{i,l,t}) + \alpha^{code}_{i,l,t} B^{code}(h_{i,l,t})$
This allows the model to allocate contributions based on the specific input instance, the network layer depth, and the current generation token, rather than using fixed global coefficients.

2. Training Framework

MoCA employs a two-stage training strategy to align structured reasoning with executable code generation:

Stage 1: Supervised Fine-Tuning (SFT) with Self-Distilled SoT
- The model is trained on paired chart-code data augmented with Snippet-of-Thought (SoT) trajectories.
- These trajectories decompose the mapping from visual input to code into explicit intermediate steps.
- Self-Distillation: Instead of using raw teacher-generated SoT data, the authors apply self-distillation to produce trajectories that better align with MoCA's own generation distribution, reducing the discrepancy between training and inference.
- The model is optimized using standard autoregressive cross-entropy loss over the combined SoT and code sequences.
Stage 2: Reinforcement Learning (RL)
- The model is further optimized using Group Relative Policy Optimization (GRPO).
- A multi-dimensional reward function is employed, aggregating five signals:
  1. Format Consistency ( $R_{fmt}$ ): Adherence to output structure.
  2. Reasoning Consistency ( $R_{con}$ ): Alignment between the SoT and the generated code.
  3. Structural Fidelity ( $R_{str}$ ): Reconstruction of chart attributes via AST analysis.
  4. Execution Validity ( $R_{exec}$ ): Whether the code runs successfully.
  5. Visual Similarity ( $R_{vis}$ ): Comparison between the rendered chart and the reference image.
- These rewards provide both process-level (reasoning) and outcome-level (execution/visual) supervision.

Key Contributions

Formulation of Adaptive Coordination: The paper frames chart-to-code generation as a cross-modal capability coordination problem, arguing that visual grounding and code generation are complementary but heterogeneous abilities that require explicit, dynamic interaction modeling.
MoCA Architecture: Introduction of a dual-branch architecture with the Cross-modal Arbitration Block (CAB). This preserves separate visual and code expertise while using a lightweight, input-conditioned Arbiter to dynamically allocate contributions at the token and layer level.
Training and Analysis Framework: Establishment of a training pipeline combining self-distilled SoT supervision with multi-dimensional RL rewards. The authors provide empirical evidence that the Arbiter learns structured allocation patterns (varying systematically across tokens, layers, and instances) rather than arbitrary or uniform weights.

Experimental Results

MoCA was evaluated on three benchmarks: ChartMimic, Plot2Code, and ChartX.

Performance: MoCA achieved state-of-the-art or competitive results across general-domain and chart-specialized models.
- On ChartMimic, it achieved the best performance across all three metrics (Execution Rate: 88.83, Low-Level: 75.13, High-Level: 73.48), outperforming the strongest baselines by significant margins.
- On Plot2Code, it achieved the highest Code Pass rate (87.88), surpassing the next-best baseline by 2.27 points.
- It maintained competitive performance on semantic metrics (Text Match) and perceptual metrics (GPT Score), often outperforming much larger models (e.g., Qwen2.5-VL-72B) in execution and reconstruction fidelity.
Ablation Studies:
- Training Stages: Both SFT and RL were shown to be necessary; SFT established a stable mapping, while RL improved executability and visual fidelity.
- Branch Initialization: A variant with two visual branches (Visual + Visual) performed worse than MoCA, confirming that gains come from complementary expertise initialization, not just parameter count.
- Arbitration: A variant with fixed branch weights (0.5/0.5) underperformed MoCA, demonstrating the necessity of input-conditioned arbitration.
- Static Merging: MoCA significantly outperformed static parameter merging approaches (like VisCodex), highlighting the limitation of fixed coefficients for heterogeneous tasks.

Significance and Claims

The paper claims that the performance gains of MoCA cannot be attributed solely to larger model size or additional training data. Instead, the improvements arise from the joint contributions of:

Complementary Branch Initialization: Leveraging pre-trained expertise in both vision and code.
Input-Conditioned Arbitration: The ability to dynamically adjust the reliance on visual vs. code branches based on the specific needs of the input and the generation context.

The authors conclude that explicitly modeling the interaction between visual grounding and code synthesis via token-level arbitration is a critical step forward for chart-to-code generation, enabling models to adaptively handle the diverse demands of different chart types. Future work is suggested to explore more parameter-efficient fusion paradigms to achieve similar integration without increasing model parameters.