🤖 machine learning

Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

This paper introduces Self-Correcting Coupled Markov Jump Processes (SC-CMJP) and the training-free $\texttt{CO}_\texttt{2}\texttt{Jump}$ sampler to enable concurrent image understanding and generation through a self-correcting mechanism that dynamically couples modalities within each step, achieving state-of-the-art performance on multimodal reasoning and editing tasks alongside the release of three new large-scale datasets.

Original authors: Minh-Quan Le, Armand Comas, Alexandros Lattas, Stylianos Moschoglou, Pedro Vélez, Amit Raj, Aaron Germuth, Thabo Beeler, Dimitris Samaras, Di Qiu

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minh-Quan Le, Armand Comas, Alexandros Lattas, Stylianos Moschoglou, Pedro Vélez, Amit Raj, Aaron Germuth, Thabo Beeler, Dimitris Samaras, Di Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a master teacher at a whiteboard. They aren't just speaking and drawing; they are doing both at the exact same time. As they sketch a curve, their words shift to describe it, and as they explain a concept, their hand adjusts the drawing to match. If they make a mistake in the sketch, they instantly erase it and redraw it while correcting their sentence. This is how human creativity often works: understanding and making are a single, tangled loop, not two separate steps.

For a long time, artificial intelligence has tried to copy this, but it usually does it clumsily. Most AI systems work like a strict assembly line: first, they write a whole story or plan, and then they try to draw a picture based on that plan. If the plan has a mistake, the picture inherits it, and the AI can't go back to fix the plan because it's already "finished" writing. This paper, titled "Concurrent Image Understanding and Generation," dives into a specific corner of AI called Masked Diffusion Models. Think of these models as a game of "Guess the Picture" where the AI starts with a blank, noisy canvas and slowly reveals the image token by token. The paper asks a big question: What if the AI could write the description and draw the picture simultaneously, letting each one correct the other in real-time, just like the teacher at the whiteboard?

The researchers introduce a new framework called Self-Correcting Coupled Markov Jump Processes (SC-CMJP). To understand this, imagine two friends trying to solve a puzzle together. In older AI methods, Friend A (the text) would shout out a guess, and Friend B (the image) would try to draw it, but they couldn't hear each other's latest thoughts until the very end. If Friend A realized they made a mistake halfway through, they couldn't tell Friend B to stop drawing that part.

This paper proposes a new way for them to talk. They call their method CO2Jump. It works like a dynamic dance where the two friends are constantly whispering to each other. If the text branch starts to commit to a description that doesn't make sense with what the image branch is seeing, the system has a special "undo" button. It can instantly "remask" (erase) a committed token and try again. This is the "Self-Correcting" part. The "Coupled" part means they don't just take turns; they weigh each other's confidence. If the text is very sure about a detail, it guides the image. If the image is very sure, it guides the text. They negotiate every single step of the process.

The team tested this on three very different challenges. First, they tried Image Editing, where the AI has to take a photo and change it based on a text prompt (like "add a hiker to this path"). They also tested Maze Solving and Nonograms (those logic puzzles where you fill in a grid based on number clues), where the text answer (the path or the filled cells) and the visual grid must match perfectly. To do this, they created three massive new datasets: JEdit-1M (1 million editing pairs), JMaze-200K (200,000 mazes), and JNono-200K (200,000 nonograms).

The results suggest that this "talking while drawing" approach works better than the old "write then draw" methods. In their experiments, the CO2Jump sampler didn't just produce better images; it also produced better text descriptions of those images. The most exciting finding is that the more steps the AI took to solve the problem, the better it got. Unlike other methods that hit a wall or get confused after a certain point, this coupled system seemed to get smarter the longer it thought, with the text and image helping each other refine the answer step-by-step. The paper shows that by letting the AI "remask" its own mistakes and listen to its other half in real-time, it can solve complex visual puzzles and edit photos with a level of coordination that was previously missing. It's a step toward AI that doesn't just follow orders, but actually understands and creates in a unified, self-correcting loop.

Technical Summary: Concurrent Image Understanding and Generation via Self-Correcting Coupled Markov Jump Processes

Problem Statement

Human cognition integrates understanding and generation in a tightly coupled loop; for instance, a teacher drawing on a whiteboard simultaneously speaks and sketches, with each modality reshaping the other. Current artificial systems, however, typically separate these processes. Even within Masked Diffusion Models (MDMs), which are theoretically suited for parallel multimodal generation, existing samplers fail to achieve true concurrency. They either decode modalities interleavedly or update them in parallel branches that share only historical state, lacking instantaneous cross-modal feedback within a single step. Furthermore, standard MDMs suffer from an inability to "remask": once a token is committed (unmasked), it cannot be revised. This leads to persistent cross-modal contradictions where text descriptions diverge from image content, or vice versa, with no mechanism for self-correction during the sampling trajectory.

Methodology

The authors introduce Self-Correcting Coupled Markov Jump Processes (SC-CMJP), a framework that models joint text-image generation as a unified stochastic process where modalities actively negotiate commitments at every denoising step.

Core Framework: SC-CMJP

SC-CMJP reformulates joint generation as a birth-death process where transition rates for one modality are functionals of the other modality's confidence.

Coupled Transition Rates: The unmasking schedule of one modality adapts to the confidence of the other. Specifically, the transition rates for image tokens depend on the text modality's unmasking confidence, weighted by cross-modal attention extracted from a single forward pass.
Remasking Jump: To address the irreversibility of standard MDMs, the framework incorporates a "remasking" jump (inspired by ReMDM). This allows committed tokens to revert to the masked state ( $\mathbf{m}$ ) if cross-modal evidence contradicts them. This transforms the process from one-way unmasking to a bidirectional birth-death process capable of retracting errors.

The CO2Jump Sampler

To instantiate SC-CMJP without architectural changes or auxiliary evaluators, the authors propose CO2Jump (Self-COrrecting COupled Jump), a training-free, single-pass sampler.

Asymmetric Scoring via Chain-Rule Decomposition: The sampler approximates the joint reverse posterior $p(z_{text}, z_{image} | z_t)$ by factorizing it. Text updates rely on self-confidence, while image updates are conditioned on the latest text state. Since a second forward pass is prohibitive, CO2Jump approximates the latest text state using the current step's hidden representations and cross-modal attention ( $A_{image \to text}$ ).
Entropy-Based Gating: A dynamic gate $\lambda$ determines the trust level in cross-modal signals. $\lambda$ is derived from the ratio of predictive uncertainties (Shannon entropy) of the two modalities. If the image is chaotic (high entropy), it defers to the text signal; if the text is uncertain, the image relies on its own self-belief.
Death and Birth Jumps:
- Death (Remasking): Tokens with the lowest coupled confidence scores are reverted to the masked state.
- Birth (Unmasking): The highest-scoring masked tokens are revealed, adjusted for the remasking quota to maintain the noise schedule.
Shared Percentile Rank: To handle vocabulary size disparities between text and image, scores are normalized to their empirical percentile ranks within each modality before mixing.

Key Contributions

Unified Stochastic Process: The paper presents the first approach to model simultaneous image understanding and generation as a single, unified stochastic process (SC-CMJP) with built-in self-correction.
CO2Jump Sampler: A novel, training-free sampler that runs in a single forward pass per step, utilizing cross-modal attention and entropy-gated coupling to resolve contradictions dynamically.
New Corpora and Benchmarks: The authors curated and will release three large-scale joint multimodal generation datasets:
- JEdit-1M: 1M image editing pairs with pixel-aligned scene graphs and reasoning traces.
- JMaze-200K: 200K maze-solving tasks with algorithmic ground truth.
- JNono-200K: 200K nonogram (Picross) puzzles.
  These include matching in-distribution and out-of-distribution (OOD) benchmarks.
Empirical Scaling: The work demonstrates that the benefits of cross-modal coupling compound across the denoising trajectory, with performance scaling monotonically with the number of steps.

Results

The authors evaluated CO2Jump against three baselines (standard MDM, ReMDM, and MMaDA-Parallel) across image editing and visual reasoning tasks.

Image Editing (JEdit-1M): CO2Jump achieved the best joint performance. On the extended ImgEditBench, it outperformed a strong sequential grounder (Qwen3-VL-8B) in grounded-understanding metrics (mAP@0.5:0.95), achieving 0.369 overall mAP compared to 0.360 for the sequential baseline. Crucially, CO2Jump was the only sampler to show monotonic improvement in both image editing quality and understanding as the number of function evaluations (NFE) increased from 8 to 512, whereas baselines plateaued or regressed.
Visual Reasoning (JMaze & JNono): CO2Jump achieved the highest joint accuracy on both in-distribution and OOD splits. Notably, on OOD grid sizes for nonograms, CO2Jump maintained performance (0.175 accuracy) while baselines dropped significantly (MDM dropped to 0.050).
Ablation Studies: Removing the "Shared Percentile Rank" caused the largest drop in editing fidelity. Removing "Entropy-Based Gating" reduced target-mAP, indicating the gate's role in late-stage sampling. Removing "Self-Correction" (remasking) caused the most significant drop in understanding (mAP), confirming that the ability to retract cross-modal contradictions is essential for joint coherence.

Significance and Claims

The paper claims that CO2Jump successfully operationalizes the "coupled loop" of human cognition for artificial systems, where text and image co-author the answer rather than being generated sequentially or independently. The primary significance lies in demonstrating that cross-modal coupling compounds across the trajectory: unlike independent samplers that drift or saturate, the coupled process allows the system to refine its solution iteratively, with text planning the edit and image executing it within the same denoising loop. The authors assert that this framework is modality-agnostic and provides a foundation for future work in audio, video, and structured outputs, though the current work is instantiated specifically on text and image.