🤖 AI

Knowledge-Centric Agents for Workflow Generation

This paper proposes a knowledge-centric framework that enhances workflow generation in visual creation systems by learning to invert, inject, and infer hierarchical knowledge from real-world examples, thereby achieving superior structural coherence and execution success rates compared to existing direct text-to-JSON approaches.

Original authors: Zhendong Li, Lei Sun, Ruibo Ming, He Zhang, Danda Pani Paudel, Luc Van Gool, Jinjin Gu

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhendong Li, Lei Sun, Ruibo Ming, He Zhang, Danda Pani Paudel, Luc Van Gool, Jinjin Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a complex machine out of LEGO bricks, but instead of snapping them together with your hands, you have to describe the entire blueprint to a robot using only words. This is the world of ComfyUI, a popular tool for creating images with artificial intelligence. In this system, every step of the creative process—like changing a color, sharpening a face, or making an image bigger—is a separate "block" or "node." To get a great result, you have to connect hundreds of these blocks in a very specific order, like a giant, intricate circuit board. If you connect the wrong two blocks, or forget to plug in a power cord (a parameter), the whole machine sputters and stops working.

For a long time, people tried to use Large Language Models (LLMs)—the same kind of smart AI that writes stories or answers questions—to automatically build these blueprints. The idea was simple: tell the AI, "Make me a picture of a steampunk airship," and have it spit out the perfect LEGO instructions. But here's the problem: these AI models are great at writing sentences, but they are terrible at building complex structures. They often forget how the blocks connect, mix up the instructions, or create blueprints that look good on paper but fall apart when you try to build them. They lack the "experience" of a human expert who knows why you connect Block A to Block B, not just that you should.

This brings us to a new paper by Zhendong Li and their team, who propose a clever new way to teach AI how to be a master builder. Instead of just asking the AI to guess the final blueprint, they teach it to think like an expert human designer. They call this a Knowledge-Centric approach. Think of it like this: if you want to teach a student how to bake a perfect cake, you don't just show them the finished cake and say, "Copy this." You teach them the strategy (why we mix eggs first), the skeleton (the list of steps without the exact measurements), and finally the recipe (the exact grams of flour).

The authors discovered that existing AI methods were failing because they tried to jump straight from the user's request to the final, messy code. To fix this, they invented a three-step process they call Knowledge Inversion, Injection, and Inference.

First, they took thousands of real, working blueprints created by humans and ran them through a "reverse engineer" process. They stripped away the messy details to find the hidden patterns. It's like taking apart a thousand different cars to figure out the general rules of how engines work, rather than just memorizing the shape of one specific car. They distilled these blueprints into three layers:

High-Level Strategy: The "big picture" plan (e.g., "We need to fix the face first, then change the style").
Skeleton Pseudo-code: The rough outline of the steps, without the specific numbers.
Full Pseudo-code: The complete, detailed instructions ready to be built.

Next, they injected this distilled knowledge into an AI model. They didn't just feed it data; they taught it to reason through these layers. They showed the model: "Here is a task, here is the strategy an expert would use, and here is how that strategy turns into a skeleton." This is like giving the student a cookbook that explains the logic of cooking, not just the recipes.

Finally, when a user asks for a new image, the AI doesn't just guess. It infers the solution by working backwards through the layers it learned. It first figures out the high-level strategy, then builds the skeleton, and finally fills in the specific details. The paper also includes a "self-refinement" step, where the AI double-checks its own work, asking, "Did I connect these blocks correctly?" before finalizing the plan.

The results are impressive. The team tested their method against other AI tools on a dataset of over 900 real-world tasks. While other methods managed to create working blueprints less than half the time, this new "Knowledge-Centric" agent succeeded in 86.9% of cases. In terms of building complex, diverse structures, their method scored a 152 on a diversity scale, while the next best method only reached 42. Even when tested on brand-new, difficult tasks that the AI had never seen before, it still managed to succeed 66.7% of the time, far outperforming the competition.

The authors suggest that by teaching AI to understand the structure and reasoning behind the work, rather than just memorizing patterns, we can create agents that are much more reliable. They admit that the system still struggles a little when asked to use very rare, unusual blocks that weren't in its training data, but overall, they have shown that giving AI a "mental model" of how experts think is the key to building better, more complex creative tools. It's a shift from asking a robot to "guess the answer" to teaching it "how to think," turning a brittle, error-prone process into a robust, expert-level craft.

Technical Summary: Knowledge-Centric Agents for Workflow Generation in ComfyUI

Problem Statement

The generation of workflows in visual creation systems like ComfyUI presents a challenge that extends beyond simple text-to-JSON translation. A ComfyUI workflow is a Directed Acyclic Graph (DAG) represented as a JSON configuration, where nodes are functional modules and edges define data dependencies. While Large Language Models (LLMs) have been applied to this task, existing approaches often fail due to two primary limitations:

Structural Brittleness: Directly generating JSON configurations from text is prone to syntactic errors, misconnected links, and inconsistent parameter bindings, leading to execution failures.
Lack of Experiential Knowledge: Effective workflow design requires expert-level reasoning, including heuristic decisions about segmentation, branching, and module interaction. Current LLMs struggle to acquire this tacit, experience-driven knowledge purely from textual data, often resulting in workflows that lack coherence or fail to address complex task requirements.

Existing methods either rely on constrained meta-nodes (limiting expressiveness) or treat knowledge as implicitly entangled within model parameters (making it difficult to interpret or reuse).

Methodology

The authors propose a knowledge-centric framework that treats workflow generation as a process of reasoning with explicit, hierarchical knowledge rather than ad-hoc pattern matching. The framework operates through three core phases:

1. Knowledge Inversion

The system first distills hierarchical knowledge representations from a large collection of raw, unannotated ComfyUI workflows. This process inverts the workflow data from the bottom up:

Raw Workflow to Full Pseudo-code: Rule-based regularization converts JSON graphs into machine-readable pseudo-code, preserving topology and parameters while removing syntactic noise.
Full to Skeleton Pseudo-code: An LLM abstracts away specific parameter bindings and numerical values, retaining only the essential graph structure and execution flow.
Skeleton to Strategy Reasoning: LLMs perform semantic interpretation to infer the "reasoning traces" behind the workflow's structure (e.g., why specific controls were fused or why a face-detailer was placed post-denoising).
Reasoning to High-Level Strategy: These traces are summarized into concise strategy descriptions representing the expert's high-level processing plan.
Strategy to Task Instruction: The system infers the original user task instructions and input specifications that likely generated the workflow, creating a natural language supervision signal.

2. Knowledge Injection

The distilled knowledge is injected into a language model (specifically Qwen3-14B) via Supervised Fine-Tuning (SFT). The authors identify specific transitions that require explicit knowledge injection:

Task $\to$ Strategy: Teaching the model to decompose user instructions into high-level processing plans and select appropriate modules.
Strategy $\to$ Skeleton Pseudo-code: Training the model to translate abstract strategies into executable structural skeletons.
Note: Transitions from Skeleton to Full Pseudo-code (parameter restoration) are handled via rule-based expansion or lightweight prompting, not SFT.
An auxiliary objective is included to train the model to generate intermediate reasoning traces, regularizing the learning of expert discourse and logical structure.

3. Knowledge Inference

During inference, the system reverses the inversion process:

High-Level Strategy Planner: The SFT-trained model generates a high-level strategy from the user's task instruction.
Pseudo-code Generator: The strategy is converted into a skeleton pseudo-code.
Parameter Filling: A pre-trained model fills in specific parameters and bindings based on context.
Rule-based Reconstruction: The full pseudo-code is converted into an executable ComfyUI JSON workflow.
To enhance robustness, self-refinement mechanisms are applied after strategy planning and pseudo-code generation, allowing the model to re-evaluate and correct structural incoherence.

Key Contributions

Knowledge-Centric Paradigm: The paper shifts the focus from direct generation to modeling the structure, hierarchy, and dynamics of expert knowledge itself.
Knowledge Inversion: A novel method to distill hierarchical representations (strategies, reasoning traces, skeletons) from raw, unannotated workflow data without manual annotation.
Hierarchical Reasoning: The framework decomposes the generation task into multi-level supervision signals (Task $\to$ Strategy $\to$ Skeleton $\to$ Full), enabling the model to capture design principles explicitly.
Reversible Reasoning & Self-Refinement: The inference process mirrors the inversion process, augmented by iterative self-correction to ensure structural coherence.

Experimental Results

The method was evaluated on a curated dataset of 912 high-quality workflows (882 for training, 30 for testing) and external benchmarks (FlowBench and ComfyBench).

Structural Diversity and Completeness: The proposed method achieved a Valid Node Diversity (VND) score of 152, significantly outperforming the strongest baseline (42). It also achieved higher Node Completeness (96.3% vs. 56.7%) and Link Completeness (98.3% vs. 56.7%).
Task Consistency: The method scored 86.9% on Task Consistency, compared to 55.1% for the best baseline (ComfyAgent).
Execution Success: On the test set, the method achieved an 86.9% Pass rate (successful execution without errors) and a 61.1% Resolve rate (successful execution fulfilling the task), compared to 36.4% and 29.4% for ComfyAgent, respectively.
Generalization: The method maintained superior performance on "Real-World Cases" (tasks with no comparable training samples) and external benchmarks, demonstrating strong generalization capabilities.
Ablation: The hierarchical approach using a 14B model outperformed an end-to-end Q2P (Question-to-Pseudo-code) baseline using a larger 32B model, validating the efficacy of the multi-level supervision strategy.

Significance and Claims

The paper claims to establish a new foundation for knowledge-driven, agentic workflow generation. By explicitly modeling and inverting expert knowledge, the framework transforms workflow generation from a brittle text-generation problem into a principled reasoning process. The authors argue that this approach overcomes the limitations of both constrained template-based systems and opaque, data-driven LLMs, enabling agents to compose complex visual pipelines with expert-level reliability and structural coherence. The work suggests that future agentic systems in visual creation should prioritize the explicit representation and inversion of domain knowledge over simple pattern completion.