💬 NLP

Crayotter: Traceable Multi-Agent Workflows for Long-Form Video Editing

Crayotter is an open-source, traceable multi-agent system that structures long-form video editing into three artifact-driven phases to enable diagnostic revisions and achieves superior performance in narrative coherence and theme alignment compared to existing baselines.

Original authors: Lecheng Yan, Yichong Zhang, Ben Pan, Xiaoyu Zheng, Jiawei Qian, Anqi Wu, Wenxi Li, Chenyang Lyu

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lecheng Yan, Yichong Zhang, Ben Pan, Xiaoyu Zheng, Jiawei Qian, Anqi Wu, Wenxi Li, Chenyang Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to make a 30-second travel video for a friend. You have a messy pile of raw footage: some clips of mountains, some of street food, some of your dog, and some blurry shots of a hotel lobby.

If you ask a standard AI to "make a travel video," it might just grab a few random clips, slap them together, and hope for the best. If the result is weird (like a shot of your dog appearing in the middle of a mountain scene), the AI usually just gives up and tries to start the whole thing over from scratch.

Crayotter is a new, open-source system that changes how AI edits videos. Instead of acting like a "black box" that magically spits out a finished product, Crayotter acts more like a human film editor with a very organized desk and a detailed notebook.

Here is how it works, broken down into simple steps:

1. The Three-Phase Workflow

Crayotter doesn't just "edit." It breaks the job down into three distinct stages, leaving a paper trail (called artifacts) at every step.

Phase 1: The Scouting Trip (Material Preparation)
Before editing, Crayotter acts like a scout. It looks at your request (e.g., "Show a vibrant culture event") and creates a checklist of what it needs: a wide shot, a close-up, a crowd scene, etc. It searches for videos that match these specific tags. If it can't find a "crowd scene," it doesn't guess; it goes back and searches again until it finds the right evidence.
- The Analogy: It's like a chef checking the pantry before cooking. If they need fresh basil and don't have it, they go to the market to get it, rather than trying to cook a pasta dish with only dried herbs.
Phase 2: The Blueprint (Editing Research)
Once the clips are gathered, Crayotter doesn't start cutting yet. It sits down and writes a blueprint. It decides the order of clips, where the music should swell, where the narrator should speak, and how long each shot should be.
- The Analogy: This is like an architect drawing a house plan before a single brick is laid. The AI creates a "script" for the video that includes timing and transitions.
Phase 3: The Construction (Tool Execution)
Now, the AI actually does the work. It uses digital tools to cut the clips, add transitions, and sync the audio. But here is the magic: it keeps a live log of every single move.
- The Analogy: Imagine a builder who takes a photo after every step. If they put a window in the wrong place, they don't have to tear down the whole house. They just look at the photo, see the mistake, and fix only that window.

2. The Superpower: "Traceability"

The biggest innovation in Crayotter is that it makes the editing process visible and fixable.

In older AI systems, if the video came out wrong, you had to say, "Do it again," and hope the AI got lucky the second time. With Crayotter, if the video fails, the system (or a human) can look at the "artifacts"—the logs, the blueprints, and the intermediate drafts—to see exactly where it went wrong.

The "Replay" Feature: You can rewind the AI's thought process. You can see, "Ah, the AI picked a clip that was too short for the narration," or "The transition between these two scenes was too abrupt."
The Fix: Instead of restarting, the AI can just go back to that specific step, swap out the bad clip, and continue. It's like editing a document where you can fix a single typo without rewriting the whole book.

3. How Well Does It Work?

The researchers tested Crayotter on 23 different video themes (like travel, campus life, pets, and food) and compared it to two other popular video-editing tools (CapCut-Mate and CutClaw).

The Results: Humans rated Crayotter significantly higher (3.40 out of 5) compared to the others (2.44 and 1.70).
Why? Crayotter was better at:
- Sticking to the theme: It didn't accidentally include random clips that didn't fit the story.
- Story flow: The video felt like a coherent story rather than a random collection of shots.
- Smoothness: The cuts and transitions felt natural.

4. The "Crayotter" Philosophy

The paper argues that for AI to be truly useful in creative tasks like video editing, it needs to stop being a "magic generator" and start being a collaborative partner.

By exposing its work (the "artifacts"), Crayotter allows humans to:

Inspect the work at any stage.
Diagnose exactly what went wrong.
Repair just the broken parts without throwing away the whole project.

In short, Crayotter turns video editing from a "roll the dice" game into a traceable, step-by-step craft where mistakes are easy to find and fix, leading to much better final videos.

Technical Summary: Crayotter

Problem Statement

Long-form video editing from heterogeneous footage presents a complex, long-horizon control problem that extends beyond simple clip selection. It requires an agent to maintain narrative intent across material preparation, timeline construction, post-production, and revision. Current agent systems often treat editing as a black-box generation task or focus primarily on synthesizing new shots rather than coordinating editing operations over real footage. This lack of formalization leads to critical issues: small errors in asset selection or timing propagate into broken continuity and unnatural pacing. Furthermore, existing workflows often lack scientific trajectory logs, making it difficult to replay, diagnose, or selectively repair failed segments without restarting the entire production.

Methodology: Crayotter Architecture

Crayotter is an open-source, multimodal multi-agent system designed for prompt-driven video editing. It decomposes the editing process into three distinct phases, externalizing state into inspectable artifacts (reports, JSONs, blueprints, tool logs, and renders) rather than relying on hidden latent states. This design enables "environment-grounded reflection," where agents and humans can diagnose failures and selectively revise specific segments.

1. Phase 1: Coverage-Aware Multimodal Footage Retrieval

The system transforms an abstract editing request into concrete visual evidence.

Tag Expansion: The user request is expanded into production-oriented tags covering scene, action, style, story, and shot types.
Iterative Search & Reranking: A coarse retriever fetches a high-recall candidate pool. These candidates are reranked based on request relevance and marginal coverage gain using sampled frames and temporal windows.
Verification: Top candidates undergo full-video verification to ensure they cover the required semantic tags. Uncovered tags trigger iterative refinement of the query until the material pool is sufficient or the retrieval budget is exhausted.
Output: A material pool $P$ containing selected sources, coverage evidence, and multimodal analysis records.

2. Phase 2: Deep Editing Research

This phase synthesizes the material pool into a time-grounded editing blueprint without executing editing tools.

Temporal Localization: To improve event localization, the system overlays human-readable time coordinates on sampled frames (time-marked views). This allows the research agent to bind semantic observations to absolute editing coordinates.
Blueprint Construction: The agent predicts source clips and specific temporal intervals for narrative beats (e.g., establishing shots, montages).
Output: An editing blueprint $B$ containing narrative structure, clip ordering, rhythm, transitions, and narration plans. This blueprint serves as a temporally addressable production plan.

3. Phase 3: Tool-Grounded Timeline Execution

The blueprint is executed via external editing tools in a ReAct-style loop.

Action Space: The agent performs actions such as cutting, merging, inserting transitions, aligning narration, generating subtitles, and normalizing audio.
Environment-Grounded Reflection: After each tool call, a verifier computes diagnostics (e.g., tool validity, timestamp accuracy, coverage preservation). If a diagnostic fails, the agent observes the specific artifact that failed and repairs only the affected segment rather than restarting the full edit.
Output: A final rendered video and a complete trace of the execution history.

Trajectory-Level RLVR

The system defines a Reinforcement Learning with Verifiable Rewards (RLVR) environment. Unlike preference-only rewards, the reward signal is grounded in observable artifacts (valid file outputs, measured durations, coverage records). The reward function decomposes into execution validity, artifact creation, alignment with the blueprint, and successful local repair. This setup supports future policy optimization using algorithms like GRPO (Group Relative Policy Optimization).

Key Contributions

Traceable Formulation: A framework where source assets, plans, timeline states, renders, and logs are explicit artifacts, enabling inspection, replay, and localized repair.
Coverage-Aware Retrieval: A multimodal footage retrieval loop that decomposes requests into coverage tags, reranks candidates via frame and temporal-window analysis, and iteratively searches for missing semantic evidence.
Open-Source System & Evaluation: The release of Crayotter as a three-phase editing system, evaluated on 23 themes against baselines (CapCut-Mate and CutClaw) with released traces exposing intermediate decisions.
Replayable Trajectory Schema: A design combining verifiable tool signals, episode-level outcome scoring, and human preference calibration to prepare workflows for future policy optimization.

Results

Crayotter was evaluated on 23 editing themes across five scenario families (pet, campus, travel, scenery, food) against CapCut-Mate and CutClaw.

Human Evaluation: Crayotter achieved an average score of 3.40/5, significantly outperforming CapCut-Mate (2.44) and CutClaw (1.70). It showed consistent gains in theme alignment, narrative coherence, and editing smoothness.
AI Evaluation: Crayotter also outperformed baselines in AI scoring (2.39 vs. 2.10 and 1.57), with the largest margins in theme alignment and narrative coherence.
Scenario Performance: The system performed strongest in scenery cases (aerial landscapes, wide shots) where rhythm maps cleanly to the tool chain. It faced more challenges in food and pet stories requiring fine-grained action coverage and strict filtering.
Process Diagnostics: Analysis of the 23 runs revealed an average of 221 observable events per run. The system successfully demonstrated localized repair (e.g., switching to a stable interval when a source interval was damaged) rather than total failure.

Significance and Claims

The paper posits that high-quality AI video editing depends not only on stronger generators but on controllable production workflows. Crayotter demonstrates that long-form editing agents can be evaluated and improved through observable production artifacts rather than latent reasoning traces alone.

By externalizing state into artifacts (coverage reports, blueprints, tool logs), the system transforms editing from a single opaque generation attempt into a reproducible trajectory of planning, execution, and repair. This approach allows for:

Diagnosis: Identifying exactly where a plan drifts or an artifact fails.
Selective Revision: Fixing specific timeline segments without restarting the entire production.
Future Optimization: Providing a verifiable reward interface for training policies via RLVR.

The authors conclude that making the editing process a "research object" through explicit artifact exposure is a critical step toward robust, long-horizon visual storytelling.