Technical Summary: Socratic-Geo

1. Problem Statement

Multimodal Large Language Models (MLLMs) have advanced significantly in vision-language understanding but remain severely bottlenecked in geometric reasoning. This limitation stems from the extreme scarcity of high-quality image-text pairs required for training. Existing solutions face critical trade-offs:

• Human Annotation: Prohibitively expensive and time-consuming.
• Automated Methods: Fail to simultaneously ensure fidelity, diversity, and training effectiveness.
  • Image-based textual augmentation is passive, refining descriptions for existing images without creating new structures.
  • Symbolic-driven random generation relies on blind exploration and heuristic filtering, decoupling generation from learning needs.
  • LLM-driven augmentation often acts as a black-box amplifier, inheriting biases and lacking fine-grained control over geometric structures.
• Static Paradigms: Most existing approaches produce one-way datasets where synthesis is decoupled from the model's learning process, missing opportunities for iterative improvement based on model weaknesses.

2. Methodology: Socratic-Geo Framework

The authors propose Socratic-Geo, a fully autonomous, dynamic framework that couples data synthesis with model learning through multi-agent interaction. Inspired by the Socratic method, the system operates as a closed-loop engine driven by three specialized agents: Teacher, Solver, and Generator.

2.1 The Teacher Agent (Proactive Constructor)

The Teacher generates parameterized Python scripts to create geometric problems, employing a "conceive-and-verify" loop to ensure purity and validity:

• Reflect (Solvability Checker): Verifies mathematical validity and solvability.
• RePI (Visual Validator): Ensures the rendered diagram aligns perfectly with the textual description and geometric axioms.
• Workflow:
  1. Diagnose: Analyzes the Solver's failure paths to identify specific reasoning weaknesses (e.g., missing axioms, semantic inconsistencies).
  2. Invent: Programmatically modifies the underlying Python code to construct a new problem that explicitly targets these weaknesses.
  3. Qualify: Self-verifies the new problem by solving it; only validated triplets are admitted to the curriculum.

2.2 The Solver Agent (Reinforcement Learning)

The Solver evolves via Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm designed for tasks with verifiable rewards.

• Training Paradigm: The Solver learns exclusively through trial-and-error. It does not observe the Teacher's chain-of-thought or reference solutions during training.
• Reward Signal: It receives binary reward signals indicating correctness. If all $k$ attempts fail, the Teacher injects the verified reference solution as the sole positive example to guide the policy update.
• Curriculum Evolution: The curriculum $C$ grows iteratively. Problems that the Solver fails to solve trigger the Teacher to invent enhanced versions, ensuring the dataset adapts to the Solver's evolving capabilities.

2.3 The Generator Agent (Synergistic Byproduct)

The Generator operates independently of the reasoning loop but leverages assets created by the Teacher.

• Distillation: The Teacher translates its programmatic drawing instructions (Python code) into natural language drawing instructions ($p_{diagram}$).
• Training: The Generator, a diffusion-based model, is fine-tuned on high-quality $(instruction, image)$ triplets. This distills the Teacher's symbolic, rule-based drawing intelligence into the Generator's neural weights, enabling it to produce geometrically precise diagrams from text prompts.

3. Key Contributions

1. Goal-Driven Programmatic Synthesis: A paradigm that establishes generation goals by diagnosing learner weaknesses and enhances image structures through code modification. This is supported by the Reflect-RePI mechanism for proactive self-correction, ensuring both mathematical and visual validity.
2. Multi-Agent Interaction Framework: Socratic-Geo tightly couples synthesis with learning. The Teacher-Solver interaction drives reasoning evolution, while the Generator extends this capability to controllable geometric image generation from programmatic data.
3. High Data Efficiency: The framework achieves state-of-the-art results starting from only 108 seed problems, demonstrating that a small, high-quality, iterative curriculum outperforms large-scale static datasets.

4. Experimental Results

4.1 Solver Performance (Geometric Reasoning)

Evaluated on six benchmarks (MathVerse, GeoQA, MathVision, MathVista, WeMath, GeomVerse):

• Performance: Socratic-Solver achieved an overall average accuracy of 49.11%.
• Comparison: This represents a +4.13 point improvement over the zero-shot baseline and surpasses the strongest baseline (GeoReasoning) by 2.43 points.
• Data Efficiency: The model achieved these results using only 2.5k synthetic training samples (one-quarter of the data used by baselines like R-CoT or GeoReasoning which use 7.2k–10k samples).

4.2 Generator Performance (Text-to-Image)

Evaluated on the GenExam-Math benchmark:

• Performance: Socratic-Generator achieved a 42.4% Relaxed Score.
• Comparison: This establishes a new state-of-the-art for open-source models, surpassing Seedream-4.0 (39.8%) and approaching the performance of the closed-source Gemini-2.5-Flash-Image (43.1%).
• Strict Score: Achieved a Strict Score of 6.0, significantly outperforming the base Qwen-Image model (0.0).

4.3 Ablation Studies

• Qualify Module: Removing the verification step led to a drop in performance (below zero-shot baseline), confirming that unverified synthetic data introduces errors that degrade learning.
• RL vs. SFT: Reinforcement Learning (RL) training outperformed Supervised Fine-Tuning (SFT) by 1.74% on identical data, proving that the trial-and-error learning paradigm is superior to imitation learning for this task.
• Teacher Capacity: Performance gains were driven primarily by the Teacher's access to Solver failure logs and code ("information privilege") rather than raw model capacity; even a 3B parameter Teacher outperformed baselines trained on 10k samples.

5. Significance and Claims

The paper claims that Socratic-Geo addresses the critical bottleneck of geometric data scarcity by shifting from static, one-way data generation to a dynamic, synthesis-learning closed loop.

• Self-Improving Engine: The framework demonstrates that a system can expand a highly informative curriculum from a minimal seed set by continuously identifying and addressing reasoning weaknesses.
• Dual Effectiveness: It successfully unifies the improvement of reasoning capabilities (Solver) with the generation of high-fidelity visual content (Generator), proving that programmatic control can bridge the gap between symbolic logic and visual representation.
• Scalability: The authors posit that this scalable, self-improving data engine can be extended to other STEM domains beyond geometry, offering a pathway to overcome data limitations in complex multimodal reasoning tasks.