🤖 AI

Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning

This paper introduces a closed-loop framework for scientific graphics program synthesis that leverages the high-quality SciTikZ-230K dataset, the comprehensive SciTikZ-Bench evaluation suite, and a novel Dual Self-Consistency Reinforcement Learning paradigm to train SciTikZer-8B, a model that achieves state-of-the-art performance in converting static visuals to executable TikZ code, outperforming leading proprietary and open-source multimodal models.

Original authors: Juekai Lin, Yun Zhu, Honglin Lin, Sijing Li, Tianwei Lin, Zheng Liu, Xiaoyang Wang, Wenqiao Zhang, Lijun Wu

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juekai Lin, Yun Zhu, Honglin Lin, Sijing Li, Tianwei Lin, Zheng Liu, Xiaoyang Wang, Wenqiao Zhang, Lijun Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, complex scientific diagram—maybe a circuit board, a flowchart for a biological process, or a physics equation drawn out. Right now, this is just a static picture (a JPEG or PNG). If you want to change a number, move a component, or fix a label, you have to redraw the whole thing from scratch. It's like trying to edit a painting by scraping off the paint and starting over.

What if you could turn that picture back into the "recipe" used to create it? In the world of science, that recipe is called TikZ. It's a programming language that tells a computer exactly where to draw every line, circle, and arrow.

The problem? Current AI models are terrible at this. They try to guess the recipe, but they often get the ingredients wrong, forget to include the oven (missing libraries), or write instructions that make no sense (syntax errors). The result? A recipe that looks okay on paper but fails to bake a cake (compilation failure).

This paper introduces a new system called SciTikZer that solves this problem. Here is how they did it, explained simply:

1. The Problem: The "Bad Chef" AI

Imagine an AI chef trying to recreate a complex dish just by looking at a photo.

The Data Gap: The AI was trained on recipes found on the internet. Many of these were messy, missing ingredients, or written in a language that didn't work in the kitchen.
The Evaluation Gap: There was no way to taste-test the dish properly. The AI could say, "I made a cake," but if the cake was raw in the middle or looked like a brick, the AI didn't know it failed.

2. The Solution: A Three-Part Super-System

Part A: The "Quality Control" Kitchen (SciTikZ-230K)

Instead of just scraping random recipes from the internet, the researchers built a Quality Control Kitchen.

The Process: They took 310,000 messy recipes and ran them through a strict test.
The Fixer Bot: If a recipe failed to compile (the "oven" wouldn't turn on), they didn't throw it away. They used a smart AI "mechanic" to read the error log, figure out what was missing (like a missing spice jar or a broken oven door), and fix it.
The Result: They ended up with 230,000 perfect, working recipes. This is the SciTikZ-230K dataset. It's like training a chef only on perfect, tested recipes instead of random blog posts.

Part B: The "Taste Test" Benchmark (SciTikZ-Bench)

To make sure their new chef is actually good, they created a rigorous Taste Test.

Instead of just asking, "Does it look like the photo?", they asked: "Is the code clean? Is the logic sound? Does it actually compile?"
They tested everything from simple shapes to complex, multi-layered scientific diagrams. This is SciTikZ-Bench.

Part C: The "Double-Check" Training (Dual Self-Consistency RL)

This is the most clever part. Imagine you are teaching a student to draw a map.

Step 1 (Visual Fidelity): You tell the student, "Draw a map that looks exactly like this photo." If the map looks blurry or the roads are in the wrong place, you give them a bad grade. This teaches the AI to pay attention to details.
Step 2 (The "Back-Translation" Trick): This is the magic. Once the student draws the map, you ask them to write a description of the map they just drew.
- The Logic: If the student truly understood the map, the description they write should match the original instructions they were given.
- The Penalty: If the student draws a weird, messy map just to get a high score, but then can't describe it back correctly, they get a massive penalty. This forces the AI to be logically consistent, not just visually lucky.

This "Draw it, then describe it, then compare" loop is called Dual Self-Consistency. It stops the AI from "cheating" (making a pretty picture that is actually nonsense code).

3. The Result: The New Champion

The result is a model called SciTikZer.

It is an 8-billion-parameter model (which is actually quite small compared to some giants).
The Performance: It beats massive models (like Google's Gemini or models with 200+ billion parameters) and specialized tools.
Why? Because it wasn't just memorizing patterns; it learned the logic of the language. It produces code that compiles 97% of the time (almost perfect) and looks exactly like the original image.

The Big Picture Analogy

Think of previous AI models as imposters. They could mimic the look of a scientific diagram but couldn't actually build it. If you asked them to change a value, they would break the whole thing.

SciTikZer is like a master architect.

It studied thousands of perfect blueprints (the curated data).
It practiced by drawing a building, then writing the instructions for that building, and checking if the instructions matched the drawing (the Dual Self-Consistency).
Now, when you show it a picture of a building, it doesn't just paint a copy; it writes the actual, working blueprint that you can edit, resize, and use forever.

This is a huge leap forward because it turns static scientific images into editable, living data, making science easier to share, understand, and build upon.

1. Problem Statement

The paper addresses the challenge of Graphics Program Synthesis, specifically the reverse-engineering of static scientific images (e.g., circuit diagrams, flowcharts, geometric proofs) into editable, executable TikZ code. While TikZ is the de facto standard for high-fidelity scientific schematics due to its programmatic flexibility, it imposes strict requirements that current Multimodal Large Language Models (MLLMs) struggle to meet:

Rigorous Spatial Precision: Unlike statistical plotting libraries, TikZ requires explicit coordinates and symbolic primitives. Minor errors lead to compilation failures or structural degeneracy.
Data Quality Gap: Existing image-TikZ datasets often suffer from non-executable code, missing dependencies, and poor visual alignment.
Evaluation Gap: There is a lack of comprehensive benchmarks that evaluate both visual fidelity and structural logic simultaneously.
Methodological Limitations: Current approaches rely on Supervised Fine-Tuning (SFT), which is rigid, or standard Reinforcement Learning (RL), which often leads to "reward hacking" (generating code that looks right but is structurally degenerate) or fails to utilize the executable nature of TikZ for closed-loop training.

2. Methodology

The authors propose a closed-loop framework comprising three core components: a high-quality data engine, a specialized benchmark, and a novel training paradigm.

A. SciTikZ-230K: Execution-Centric Data Engine

To overcome data quality issues, the authors developed an automated pipeline to curate SciTikZ-230K, a dataset of 230,000 high-fidelity image-code pairs covering 11 scientific disciplines.

Active Remediation: Instead of discarding noisy web-scraped data, the system uses an MLLM (Qwen3-VL) to refactor non-standalone code and repair compilation errors based on compiler logs (e.g., adding missing libraries like circuitikz or backgrounds).
Coarse-to-Fine Purification:
- Heuristic Sanitization: Filters out samples with excessive length, redundancy, or external dependencies.
- Fidelity Adjudication: Uses an MLLM-as-a-Judge to score samples on Correctness, Layout, Readability, Scientific Plausibility, and Visual Complexity. Only samples passing strict multi-criteria thresholds are retained.

B. SciTikZ-Bench

A new evaluation benchmark containing 611 manually verified, decontaminated samples ranging from basic geometric primitives to complex hierarchical schematics. It evaluates models across visual fidelity (SigLIP, LPIPS, SSIM) and code quality (CrystalBLEU, Token Edit Distance).

C. SciTikZer: Dual Self-Consistency (DSC) Reinforcement Learning

The core algorithmic contribution is a two-stage RL paradigm designed to align visual output with symbolic logic.

Stage 1: Visual Fidelity Alignment:
- Uses Group Relative Policy Optimization (GRPO).
- Employs an Execution-Gated Reward: A hard penalty for compilation failures and a positive reward for successful rendering.
- Uses a Multi-Granularity Visual Reward: Combines semantic alignment (SigLIP) and structural precision (LPIPS) to ensure the rendered image matches the ground truth pixel-perfectly.
Stage 2: Self-Consistency Refinement (The Novelty):
- Introduces a Round-Trip Verification mechanism inspired by dual learning.
- Process: The model generates code $y$ from image $I \to$ renders image $\hat{I} \to$ the model reconstructs code $y'$ from $\hat{I}$ .
- Reward: A consistency reward is applied only if the visual fidelity exceeds a threshold. The reward is based on the similarity between the original code $y$ and the reconstructed code $y'$ (using Token Edit Distance and CrystalBLEU).
- Goal: This forces the model to generate "canonical" code that is not only visually accurate but also logically self-consistent, preventing degenerate solutions that hack the visual reward.

3. Key Contributions

SciTikZ-230K Dataset: A large-scale, strictly compilable, and visually aligned dataset constructed via an execution-centric data engine, significantly reducing noise and hallucination in training data.
SciTikZ-Bench: A rigorous, multi-dimensional benchmark for evaluating scientific graphics synthesis, covering diverse domains and difficulty levels.
Dual Self-Consistency (DSC) RL: A novel optimization paradigm that unifies visual fidelity and structural logic through round-trip reconstruction, effectively penalizing degenerate code and improving logical robustness without relying on ground-truth code for the RL phase.
SciTikZer-8B: An open-source model that achieves State-of-the-Art (SOTA) performance, outperforming proprietary giants (Gemini-2.5-Pro, GPT-5) and massive open-source models (Qwen3-VL-235B) despite having significantly fewer parameters (8B).

4. Experimental Results

Performance on SciTikZ-Bench: SciTikZer-8B achieved a 97.2% compilation success rate, surpassing the 92.1% of Qwen3-VL-235B and 88.9% of Gemini-2.5-Pro.
Visual Fidelity: It achieved the highest SigLIP (93.8) and lowest LPIPS (29.7) scores, indicating superior semantic and structural alignment.
Generalization: On the external DaTikZ-v3 test set, SciTikZer-8B maintained dominance with a 94.46% compilation rate and the lowest distribution gap (KID: 1.14), proving it learned generalizable syntax rather than overfitting.
Ablation Studies:
- Data Curation: Training on the curated SciTikZ-230K improved compilation rates by ~5% and SigLIP by ~5 points compared to raw data.
- DSC Effect: Adding Stage 2 (DSC) further improved compilation rates and reduced LPIPS, confirming that self-consistency constraints prevent visual hacking and improve structural logic.
Human Evaluation: In blind tests, SciTikZer-8B was preferred 59% of the time over baselines, with the highest scores in Visual Fidelity, Structural Correctness, and Code Quality.
Cross-Language Generalization: The DSC framework was successfully applied to Python code generation (ChartMimic), demonstrating its versatility beyond TikZ.

5. Significance

This work represents a significant leap in executable visual program synthesis. By shifting from passive data scraping to an execution-centric approach and introducing Dual Self-Consistency, the authors solve the critical trade-off between visual appearance and structural validity.

Practical Impact: The ability to generate error-free, editable TikZ code from images enables scientists and engineers to rapidly reverse-engineer and modify complex diagrams, accelerating research workflows.
Methodological Impact: The DSC paradigm offers a new blueprint for training MLLMs on tasks where the output is executable code, demonstrating that closed-loop verification (render $\to$ reconstruct) is a powerful mechanism for enforcing logical consistency in generative models.
Open Science: The release of the dataset, benchmark, and model (SciTikZer) provides a standardized foundation for future research in scientific graphics and multimodal reasoning.