Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designs

This paper introduces Feynman, a scalable knowledge-infused agent that generates high-quality, well-aligned diagram-caption pairs using an iterative code-planning and optimization-based rendering pipeline, resulting in a 100k+ dataset and a new benchmark called Diagramma for evaluating visual reasoning in vision-language models.

The Gist

Imagine you want to teach a robot how to draw a perfect map of a city, but the robot is terrible at two things: knowing the facts (like where the library is) and drawing the lines (making the streets look neat).

Most current AI models are like a brilliant art student who has never studied geography, or a geography professor who can't hold a pencil. They either draw beautiful but factually wrong maps, or they write perfect descriptions of cities that look like messy scribbles when drawn.

This paper introduces FEYNMAN, a new AI agent that solves this problem by acting like a super-efficient construction crew rather than a single artist. Here is how it works, broken down into simple steps:

1. The Problem: The "All-in-One" Trap

Usually, when you ask an AI to "draw a diagram of a chemical reaction," it tries to do everything at once: think of the chemistry, decide what the atoms look like, and draw the picture.

• The Result: It often gets the chemistry wrong, or the picture looks messy and unreadable. It's like asking a chef to design the menu, cook the meal, and plate it all in one second. Something usually goes wrong.

2. The Solution: The FEYNMAN Assembly Line

FEYNMAN changes the game by breaking the job into three distinct roles, passing the work down the line like a relay race.

Step 1: The "Idea Generator" (The Architect)

First, FEYNMAN asks a smart AI (like a human expert) to just list the facts.

• Analogy: Imagine an architect sitting down and saying, "Okay, for this house, we need a kitchen, a living room, and a roof. The kitchen needs a stove."
• FEYNMAN doesn't draw anything yet. It just writes down the knowledge (the "ideas") needed for the diagram. This ensures the science or math is 100% correct before a single line is drawn.

Step 2: The "Translator" (The Blueprint Maker)

Next, FEYNMAN takes those facts and translates them into a special language called "Substance."

• Analogy: The architect hands the list to a translator who speaks "Blueprint." The translator writes: "Place the stove here. Put the table there."
• Crucially, this language doesn't say exactly where to put the pixels. It just says what needs to be there and how they relate to each other.

Step 3: The "Artistic Builder" (The Renderer)

This is where the magic happens. FEYNMAN uses a tool called PENROSE.

• Analogy: Imagine a robot builder who has a blueprint. Every time you ask it to build the house, it follows the rules (kitchen next to living room) but randomly decides the exact shade of the walls, the angle of the roof, or the size of the windows.
• Because the rules are strict but the details are random, PENROSE can generate 100 different versions of the same diagram. They all mean the same thing, but they all look different. This gives us "visual diversity" without losing the "knowledge accuracy."

Step 4: The "Quality Control" (The Panel of Judges)

Before FEYNMAN saves the diagram, it shows it to a panel of other AIs (the "judges").

• Analogy: Think of a reality TV show where the judges critique the house. "Hey, that stove is too close to the window," or "The text is too small to read."
• If the judges say it's bad, FEYNMAN goes back, fixes the code, and tries again. It keeps doing this until the diagram is perfect.

3. The Big Win: A Massive Library of Diagrams

Because this process is automated and efficient, FEYNMAN didn't just make a few diagrams. It built a giant library of over 100,000 high-quality diagrams covering math, science, and computer science.

• It cost less than $400 to make this library (using a cheap AI model for the heavy lifting).
• It created a new test called DIAGRAMMA to see if other AI models can actually understand these diagrams. The results? Even the smartest AIs today struggle with them, proving that diagrams are still a hard challenge for machines.

Why This Matters

Think of FEYNMAN as the industrial revolution for educational diagrams.

• Before: A teacher had to spend hours drawing a diagram by hand, or an AI would draw a messy, confusing one.
• Now: FEYNMAN can instantly generate thousands of clear, accurate, and diverse diagrams for textbooks, websites, and AI training.

It separates the brain (the knowledge) from the hand (the drawing), allowing both to do what they do best. This means we can finally teach AI to "see" and "understand" complex visual concepts, just like a human does.

Technical Summary

1. Problem Statement

The paper addresses the critical bottleneck in training state-of-the-art Multi-Modal Large Language Models (MLLMs): the scarcity of high-quality, knowledge-rich, and well-aligned image-text pairs, specifically for conceptual diagrams.

• Current Limitations: Existing MLLMs struggle to understand, use, or generate simple visual objects found in diagrams (e.g., graphs, chemical structures). They often fail at rudimentary vision tests, perceive graph structures poorly, and lack compositional understanding of visual attributes.
• Synthesis Challenges: Current methods for synthesizing vision-language data are insufficient:
  • Diffusion Models: Generate visually pleasing raster images but often hallucinate details, ignore specific concepts, or fail to render legible text.
  • Program Synthesizers (e.g., TikZ): Struggle with the complexity of coding languages, leading to low compilation success rates and requiring expensive tree-search methods to generate simple diagrams.
  • Monolithic Approaches: Asking a single model to perform knowledge elicitation (identifying concepts) and visual production (drawing the image) simultaneously results in poor quality in either domain.

2. Methodology: The FEYNMAN Agent

The authors propose FEYNMAN, a scalable agent pipeline that decouples knowledge elicitation from visual production. The system leverages the reasoning capabilities of LLMs for knowledge and the optimization-based rendering engine of PENROSE for visual diversity.

The pipeline consists of four distinct steps:

A. Idea Step (Knowledge Elicitation)

• Process: An LLM is prompted to enumerate domain-specific knowledge components ("ideas") based on a selected domain (e.g., "List N pedagogical chemical reactions").
• Goal: To isolate abstract knowledge generation from visual constraints, ensuring the content is scientifically accurate and diverse before any visual planning begins.

B. Plan Step (Code Planning)

• Process: For each generated idea, the agent devises a plan to translate concepts into PENROSE Substance code.
• Mechanism: The agent is provided with PENROSE documentation and in-context examples. It outlines the visual elements and relationships required but does not write the executable code yet. This intermediate planning step serves as a crucial foundation for successful code generation.

C. Iterate Step (Iterative Visual-Refine)

• Process: The agent translates the plan into compilable PENROSE code.
• Visual Judges: The generated code is compiled into images. These images are evaluated by a panel of visual judges (a set of MLLMs acting as critics).
• Feedback Loop: The judges assess the diagrams based on criteria such as correct representation, legible text, and proper relationships. They provide boolean feedback and suggestions.
• Refinement: If the score is below a threshold or compilation fails, the agent receives the feedback and iterates to refine the code. This process is termed Iterative Visual-Refine.
• De-duplication: To ensure diversity, the system filters out Substance codes that are too similar (using Levenshtein distance on statements) to prevent the agent from generating redundant diagrams.

D. Render Step (Optimization-Based Rendering)

• Tool: The final Substance code is rendered using the PENROSE system.
• Mechanism: PENROSE separates the Substance (abstract concepts) from the Style (visual representation). It formulates diagram layout as a constrained optimization problem, solving for shape positions and sizes that satisfy geometric constraints while minimizing an energy function.
• Benefit: This allows a single set of concepts (Substance) to be rendered into multiple diverse visual layouts (different colors, positions, styles) while preserving semantic consistency.

E. Data Generation

• QA Pairs: For every generated diagram, the system automatically creates grounded captions and multiple-choice questions (QA) covering various visual reasoning skills (e.g., spatial relationships, logical reasoning, arithmetic).

3. Key Contributions

1. FEYNMAN Agent: A novel agent architecture that decouples knowledge planning from visual production, achieving a high yield rate of textbook-quality diagrams.
2. Scalable Dataset: The generation of 106,930 well-aligned diagram-caption pairs from 10,693 unique knowledge-infused programs. This was achieved cost-effectively (under $400 using GPT-4o-mini) with a total token usage of ~1.55 billion.
3. DIAGRAMMA Benchmark: A new, fresh visual reasoning benchmark containing 1,058 multiple-choice questions across 6 subjects (Math, CS, Science, etc.). Unlike existing benchmarks, these diagrams do not exist on the internet, preventing data contamination.
4. Insights on Agent Design: Comprehensive ablation studies demonstrating that explicit knowledge planning and code planning are essential for high-quality, diverse diagram generation.

4. Results

• Benchmark Performance (DIAGRAMMA):
  • Evaluated 17 state-of-the-art MLLMs (including GPT-4o, Claude-3.5, Gemini-1.5, and open-source models like Llama-3.2-VL).
  • Top Performer: Claude-3.5-Sonnet achieved the highest overall accuracy (59.64%).
  • Difficulty: Computer Science (graph reasoning) remained the most difficult domain for all models.
  • Observation: Larger models generally performed better, confirming the benchmark's validity. Interestingly, the cheaper Gemini-1.5-Flash outperformed the more expensive Gemini-1.5-Pro, likely due to the "freshness" of the data reducing overfitting or rejection rates.
• Ablation Studies:
  • The full pipeline (Knowledge Planning + Code Planning + Early Stop) achieved the best average judge scores (65.4) and the highest yield rate (82.5%).
  • Removing Knowledge Planning (KP) significantly reduced the diversity and success rate, highlighting the necessity of separating knowledge from visual coding.
• Comparison with Baselines:
  • Compared against AUTOMATIKZ (TikZ generator) and FLUX-Pro (Diffusion model).
  • FEYNMAN significantly outperformed baselines in generating legible text and accurate concept representation. Diffusion models failed to render text correctly, and AUTOMATIKZ struggled with compilation success and layout diversity.

5. Significance and Future Work

• Scalability: FEYNMAN provides an economical and scalable solution for generating high-quality scientific diagrams, a task previously too expensive or complex for automated pipelines.
• Data for AI: The released dataset and benchmark (DIAGRAMMA) offer a crucial resource for training and evaluating the visual reasoning capabilities of next-generation MLLMs.
• Open Source: The authors plan to release the dataset, benchmark, and the full agent pipeline.
• Limitations & Future Directions:
  • Knowledge Dependency: The system relies on the implicit knowledge of the underlying LLM; future work may integrate Retrieval-Augmented Generation (RAG) for domains where LLMs have gaps.
  • Style Control: While layout diversity is high, control over specific aesthetic styles (color schemes) is currently limited to PENROSE's default capabilities. Future work aims to systematically vary Style and Domain programs.

In summary, FEYNMAN represents a paradigm shift in synthetic data generation for visual reasoning, moving from monolithic image generation to a structured, knowledge-first, code-driven approach that ensures both semantic accuracy and visual diversity.