🤖 machine learning

Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning

This paper introduces CoCoEvolve, a self-supervised framework that enhances cross-representation understanding among charts, tables, and code by defining explicit one-to-one correspondences and optimizing for consistency through a co-evolutionary process during both training and inference, thereby eliminating the need for costly annotations while improving performance across multiple benchmarks.

Original authors: Xuehang Guo, Pengyuan Li, Tom Hope, Tirthankar Ghosal, Manling Li, Qingyun Wang

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuehang Guo, Pengyuan Li, Tom Hope, Tirthankar Ghosal, Manling Li, Qingyun Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world, but you only give it three different languages to describe the same thing: a picture of a graph, a spreadsheet of numbers, and the computer code used to draw the graph. This is the challenge of "cross-representation learning." In the real world, data isn't just one thing; it's a chart you see, a table you read, and the instructions a computer follows to make them. The problem is that these three things don't always match up perfectly. One picture might be drawn from many different spreadsheets, and one spreadsheet could be turned into many different codes. It's like trying to translate a poem into a song and a painting, but you don't have a dictionary, and the translator keeps guessing.

For a long time, scientists tried to fix this by hiring humans to write down the "perfect" translation for every single example. But this is expensive, slow, and often impossible because there isn't just one right answer. The big question has been: Can we teach computers to learn these connections without needing a human teacher for every step? Can we get the computer to check its own work by seeing if the picture, the numbers, and the code all tell the same story?

This is where a new idea called CoCoEvolve comes in. Think of it as a game of "telephone" played by three friends who are trying to describe the same secret object. In this game, one friend draws the object, another writes down the measurements, and the third writes the instructions to build it. Usually, if they make a mistake, the story falls apart. But CoCoEvolve changes the rules. Instead of waiting for a teacher to say "Good job" or "Try again," the three friends constantly check each other. If the drawing doesn't match the measurements, or the instructions don't build the right picture, they know something is wrong. They use this "mismatch" as a signal to improve, without ever needing a human to tell them the answer.

The paper introduces a system where two AI models act as these friends. One model tries to turn a chart image into a table and code, while the other tries to turn that table back into code. They work in a cycle, constantly evolving together. If the code they generate can actually run and draw a chart that looks like the original image, they get a "reward." If the code fails or the chart looks weird, they learn from that failure. The researchers found that by forcing the AI to keep the picture, the numbers, and the code in perfect agreement, the models got much better at understanding data.

The team tested this on four different benchmarks (sets of test questions) and found that CoCoEvolve significantly improved the AI's performance. In some cases, the models got up to 37.91% better at translating charts to code on new, unseen data, and up to 46.88% better in other complex scenarios. Even more impressively, the system worked without needing any extra human labels to teach it. It just used the consistency between the different forms of data as its own teacher.

The paper also argues against the old way of doing things, which assumed that every chart had exactly one perfect table and one perfect code. The researchers showed that this "one-to-one" assumption is actually a trap. Because one chart can be described in many valid ways, forcing the AI to pick just one "correct" answer often confuses it and makes it perform worse. By embracing the idea that there are many valid ways to describe the data, but that all valid ways must agree with each other, CoCoEvolve avoids these traps.

In short, the paper suggests that instead of hiring a army of teachers to grade every single homework assignment, we can teach AI to be its own teacher by making sure its different "languages" (images, tables, and code) tell a consistent story. The results suggest this method is a powerful, cheaper, and more flexible way to help AI understand the complex, multi-faceted world of data.

Technical Summary: CoCoEvolve

Problem Statement

The paper addresses the fundamental challenges in cross-representation understanding across three modalities: chart images ( $V$ ), tabular data ( $T$ ), and visualization code ( $C$ ). Current AI systems struggle with this task due to three primary bottlenecks:

Inherent One-to-Many Ambiguity: A single chart image can correspond to multiple valid tabular representations and multiple valid rendering programs. Existing benchmarks often assume unconstrained one-to-one ground truths, which are ill-defined and fail to capture the natural redundancy and ambiguity of these mappings.
Supervision Ambiguity and Cost: Accurately labeling the vast space of potential correspondences is prohibitively expensive, leading to a scarcity of high-quality, principled annotations.
Lack of Principled Optimization Signals: Existing methods rely on fixed, task-specific objectives (e.g., chart-to-table or chart-to-code). These local objectives fail to enforce global semantic correctness and do not generalize to reversed directions (e.g., table-to-chart) or unseen representation combinations (e.g., table-to-code).

Methodology: CoCoEvolve

The authors introduce CoCoEvolve, a consistency-driven co-evolution framework designed to optimize models without additional annotations by leveraging the mutual agreement between representations.

1. Principled Constraint Definition

To resolve the one-to-many ambiguity, the framework grounds one-to-many mappings to precise one-to-one correspondences by augmenting the mapping functions with auxiliary descriptive constraints ( $s \in S$ ). This grounds the mappings such that a chart image $v$ , a table $t$ , and a code snippet $c$ are treated as multiple representations of the same underlying semantics, conditioned on specific constraints.

2. Co-Evolution Framework

The framework employs two models, $M_\theta$ (handling $V \to T$ and $V \to C$ ) and $M_\psi$ (handling $T \to C$ ), which co-evolve through a representation cycle ( $V \to T \to C \to V$ ).

CoCoEvolve@Train (Training-Time): The models are trained jointly on unannotated chart images. Instead of relying on ground-truth labels, the training signal is derived from cross-representation consistency.
- Hierarchical Rewards: The system computes a shared reward based on:
  - Code Consistency ( $F_c$ ): Measures semantic agreement between code generated by both models via execution success and embedding similarity.
  - Visual Consistency ( $F_v$ ): Measures visual agreement between rendered charts using four metrics: CLIP semantic similarity, SSIM structural similarity, OCR textual accuracy, and DINOv2 perceptual similarity.
  - Format Reward ( $F_f$ ): Ensures well-formed structured outputs.
- Coupled Optimization: The models are coupled through sampling and reward. $M_\psi$ conditions on the output of $M_\theta$ , and the advantage for each model is derived from a reward functional that depends on the other model's output. This prevents degenerate solutions where models collude to produce mutually consistent but semantically incorrect outputs by incorporating individual grounding rewards against the original chart.
CoCoEvolve@Test (Test-Time): At inference, the framework applies the same consistency objective to co-optimize the models. Given an input, it generates multiple rollouts from both models and selects the best output based on the consistency rewards, effectively refining the prediction at inference time without updating the model weights.
Supervised Extension: The framework can seamlessly integrate a teacher module ( $M$ ) when high-quality annotations are available. This allows for a linear annealing strategy where the system transitions from teacher-guided co-training to fully self-supervised co-evolution.

3. Evaluation Suite (CoCoEvolve@Eval)

The authors propose a unified evaluation suite that avoids the unreliability of collapsing complex assessments into single scores. It decomposes evaluation into interpretable, multidimensional metrics:

Rule-as-Judge: Deterministic metrics for structural, semantic, perceptual, and textual accuracy in charts; executability and code quality in code; and schema/value accuracy in tables.
LLM-as-Judge & MLLM-as-Judge: Fine-grained scoring (0-5 scale) across dimensions like data correctness, chart type, structural fidelity, and visual style, rather than a single wide-range score.

Key Contributions

Principled Constraint Definition: The paper introduces a formal definition to ground inherent one-to-many mappings to precise one-to-one correspondences, addressing a gap overlooked by existing benchmarks.
CoCoEvolve Framework: A novel co-evolution framework that transforms cross-representation consistency into a principled, annotation-free self-supervision signal. It enables optimization at both training and test times, agnostic to task directions and representation combinations.
Unified Evaluation Suite: A systematic evaluation suite covering all six cross-representation tasks in the cycle, utilizing multidimensional assessments to overcome the limitations of current LLM/MLLM-based judging approaches.
Empirical Validation: Extensive experiments demonstrating significant performance gains across multiple benchmarks and model architectures.

Results

Experiments were conducted on four benchmarks (ChartCoder, ChartNet, ChartMimic, Chart2Code) using various MLLMs and LLMs (e.g., Qwen3-VL, InternVL3.5, DeepSeek-Coder).

Performance Gains: CoCoEvolve consistently improves performance in both training-time and test-time settings.
- On non-overlapping test sets, it yields up to 37.91% gains.
- In out-of-domain settings, improvements reach up to 46.88%.
- Code execution success rates improved significantly, with some configurations achieving 100.00% sandbox execution success.
Generalizability: The method demonstrates robust generalization to out-of-domain tasks (varying complexity and domains like physics, finance, biology) and different reinforcement learning algorithms (GRPO, DAPO, GSPO).
Ablation Studies: The inclusion of a teacher module at both training and test times yields the most balanced performance, while test-time optimization alone consistently improves results across all six tasks.

Significance and Claims

The paper claims that CoCoEvolve offers a scalable alternative to annotation-heavy pipelines for structured reasoning tasks. By treating cross-modal consistency as a reliable supervision signal, the framework enables models to "co-evolve symbiotically," where the improvement of one model provides stronger supervision for the other. The authors assert that this approach effectively addresses the ambiguity of supervision and the lack of direction-adaptive optimization signals, leading to enhanced cross-representation understanding capabilities that generalize across tasks, directions, and domains. The work suggests that enforcing global semantic correctness through consistency is a viable path forward for multimodal reasoning without relying on fixed ground-truth labels.