💻 computer science

Self-Consistency for LLM-Based Motion Trajectory Generation and Verification

This paper adapts the self-consistency technique to the visual domain by modeling motion trajectories as prototype shapes paired with geometric transformation groups, enabling an LLM-based system to generate and verify motion graphics with significantly improved accuracy through trajectory clustering and consistency checking.

Original authors: Jiaju Ma, R. Kenny Jones, Jiajun Wu, Maneesh Agrawala

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaju Ma, R. Kenny Jones, Jiajun Wu, Maneesh Agrawala

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director asking a very talented, but slightly scatterbrained, robot assistant to draw a specific path for a character to walk on. You say, "Make the character walk in a spiral."

The robot is eager to please, but it doesn't have a single "spiral" in its brain. Instead, it has a million different ideas of what a spiral could look like. It might draw a tight spiral, a loose one, a spiral that's tilted sideways, or one that's stretched out like a spring.

Sometimes, the robot gets it right. Sometimes, it draws a circle by mistake. Sometimes, it draws a spiral that looks like a pretzel. If you ask the robot to do this just once, it might give you a pretzel, and you'd have to say, "No, that's not a spiral!"

This paper introduces a clever trick called "Self-Consistency" to fix this problem, specifically for motion graphics (animations where things move in shapes).

The "Crowd Wisdom" Strategy

Instead of asking the robot for one answer, the researchers tell it: "Give me 20 different versions of a spiral path."

Now, the robot produces a messy pile of 20 drawings. Some are perfect spirals, some are circles, and some are weird squiggles.

Here is the magic part: The correct answer is usually the one that shows up the most often.

If 15 of the drawings look like spirals (just slightly different sizes or rotations) and only 5 look like circles, the researchers say, "Okay, the robot knows what a spiral is; it just got a bit confused on the details. Let's pick the average of those 15 spiral drawings."

This is the core of Self-Consistency: By asking for many guesses and finding the "majority vote," you get a much better result than asking for just one.

The Problem: How do you count the votes?

In a math problem, counting votes is easy. If the robot says "42," "42," "42," and "100," you just count the numbers. "42" wins.

But in visual art, it's harder.

Robot A draws a spiral that starts at the bottom.
Robot B draws a spiral that starts at the top.
Robot C draws a spiral that is rotated 45 degrees.

Are these the same? To a human, yes. To a computer, they look totally different. If you just compare them pixel-by-pixel, they all look like different answers, and you can't find a "majority."

The Solution: The "Shape Family" Concept

The researchers solved this by teaching the computer to understand geometric transformations. They realized that a "spiral" isn't just one specific drawing; it's a family of shapes.

Think of it like a clay sculpture.

If you have a clay spiral, you can rotate it, shrink it, stretch it, or flip it.
Even though the clay looks different after you twist it, it's still the same shape family.

The paper creates a "hierarchy" of rules (like a set of tools) to decide what counts as the "same shape":

Rigid: You can only move it or rotate it (like a solid rock).
Similarity: You can move, rotate, and change the size (like a rubber stamp).
Affine: You can move, rotate, stretch, and squash it (like a piece of taffy).

The computer tries to group the robot's 20 drawings into families using these rules. It asks: "Can I turn Drawing A into Drawing B just by stretching it?" If yes, they are in the same family.

The "Detective" Criteria

Since the computer doesn't know which rule (Rigid, Similarity, or Affine) the robot was thinking of, it uses two detective strategies to figure it out:

The "Majority Rule" Detective: It tries the strictest rules first. If it can't find a big group of similar drawings, it relaxes the rules (allows stretching) until it finds a big group. It assumes the robot is mostly right, so the biggest group must be the correct shape.
The "Consistency" Detective: It starts with the loosest rules (allowing any stretching) and sees if the big group stays together. If the group breaks apart when you get stricter, it knows it went too far. It stops at the point where the group is still solid.

Why This Matters

The researchers tested this on over 200 different prompts (like "draw a pentagon," "draw a figure-8," "draw a parabola").

Generation: When they used this method to create animations, the robot got the shape right 4–6% more often than if they just asked for one answer.
Verification: When they used this to check if an animation was correct (e.g., "Is this a spiral?"), it was 11% more accurate than using other advanced AI models that just "look" at the picture.

The Bottom Line

This paper is like giving a confused artist a second chance. Instead of accepting their first, shaky sketch, you ask for a whole gallery of sketches. Then, you look for the theme that repeats the most, ignoring the weird outliers.

By understanding that "a spiral" can look many different ways (big, small, tilted), the computer learns to ignore the noise and find the true, consistent answer hidden in the crowd. It's a way to make AI smarter, not by teaching it more facts, but by teaching it how to trust the crowd.

1. Problem Statement

Large Language Models (LLMs) have demonstrated the ability to generate code for motion graphics animations based on natural language prompts (e.g., "Move the circle in a spiral path"). However, these models often fail to strictly adhere to the geometric specifications of the prompt, producing trajectories that deviate from the intended shape family.

The core challenge is twofold:

Generation: How to select the most accurate trajectory from a set of diverse LLM-generated samples without ground-truth supervision.
Verification: How to automatically verify if a new trajectory matches a text prompt.

In natural language reasoning, "self-consistency" (sampling multiple outputs and taking a majority vote) works because answers can be compared via identity matching (e.g., the number "42"). In visual domains like motion trajectories, exact pixel-level or coordinate-level matching is impossible due to the inherent ambiguity of natural language prompts (e.g., a "spiral" can be any size, rotation, or position). Therefore, a new definition of "consistency" is required that accounts for geometric transformations.

2. Methodology

The authors propose a framework that adapts self-consistency to visual motion trajectories by modeling prompts as Shape Families defined by Lie Transformation Groups.

A. Shape Families and Transformation Groups

Instead of treating a prompt as a single trajectory, the method models it as a family of shapes $\mathcal{F}(o, W)$ , where:

$o$ is a prototype trajectory.
$W$ is a transformation group (warp) that defines allowable variations (e.g., translation, rotation, scaling, reflection, shear).
Two trajectories are considered "consistent" if one can be transformed into the other using a warp $w \in W$ .

The authors utilize a hierarchy of Lie groups, ranging from restrictive to permissive:

Rigid ($SE(2)$): Translation + Rotation.
Rigid + Reflections ( $E(2)$ ): Adds reflection.
Similarity ( $Sim^+(2)$ ): Adds uniform scaling.
Similarity + Reflections ($Sim(2)$): Adds reflection.
Anisotropic Similarity: Adds non-uniform scaling (without shear).
Affine ($Aff(2)$): Adds shear and non-uniform scaling.

B. Warp-Invariant Distance Metric

To cluster trajectories, the paper defines a distance metric $d_W(t_1, t_2)$ that is invariant under the transformation group $W$ .

The metric is computed using a generalized Iterative Closest Point (ICP) algorithm.
It finds the optimal warp $w \in W$ that minimizes the point-to-point distance between two trajectories.
If $d_W(t_1, t_2) < \tau$ (a threshold), the trajectories are considered members of the same cluster.

C. The Self-Consistency Pipeline

Sampling: The LLM is prompted to generate $N$ diverse motion trajectory programs (SVG/JS) based on a text prompt.
Clustering: The sampled trajectories are clustered using DBSCAN under different transformation groups from the hierarchy.
Decision Criteria (Selecting $W$ ): Since the correct transformation group is unknown, the system uses two unsupervised criteria to select the best group:
- Majority-Consensus: Iterates from the most restrictive group to the least. It selects the first group where a cluster contains a strict majority (>50%) of samples. This assumes the LLM generates correct trajectories more frequently than incorrect ones.
- Hierarchical-Consistency: Iterates from the most permissive group to the most restrictive. It starts with the largest cluster under a permissive warp and checks if the cluster "breaks apart" as constraints tighten. It selects the most restrictive group where the largest cluster remains intact. This assumes incorrect trajectories will not cluster with correct ones under any warp.
Output:
- Generation: The centroid of the largest cluster is chosen as the final self-consistent trajectory.
- Verification: A query trajectory is verified by checking if its distance to the cluster centroid (under the selected $W$ ) is below the threshold $\tau$ .

3. Key Contributions

Visual Self-Consistency: Extends the self-consistency paradigm from discrete text/numbers to continuous visual domains by defining consistency via geometric transformation groups rather than identity matching.
Unsupervised Shape Family Recovery: Proposes a method to automatically recover the correct geometric transformation group (and thus the shape family) from LLM samples without ground-truth labels, using hierarchical clustering analysis.
Benchmark Creation: Introduces a new benchmark of 224 prompts covering 35 geometric base shapes (e.g., spirals, polygons, waves) with ground-truth shape families, and a verification test set of 2,240 trajectory-prompt pairs.
Performance Gains: Demonstrates that this unsupervised approach significantly outperforms direct LLM generation and Visual Language Model (VLM) baselines.

4. Results

The method was evaluated on the constructed benchmark using GPT-4.1 and GPT-5.

Motion Trajectory Generation:
- Baseline: Direct LLM generation (single sample) achieved 62.1% (GPT-4.1) and 79.1% (GPT-5) accuracy.
- Self-Consistency: Improved accuracy by 4–6%.
  - With Majority-Consensus: 68.0% (GPT-4.1) and 83.3% (GPT-5).
  - With Hierarchical-Consistency: 66.7% (GPT-4.1) and 82.6% (GPT-5).
- The "Oracle" setting (knowing the correct $W$ ) achieved 68.5% and 83.5%, showing the unsupervised criteria nearly match the theoretical optimum.
Motion Trajectory Verification:
- Baseline: VLMs (GPT-4.1/5) acting as verifiers achieved F1 scores of ~75.6% and 79.0%.
- Self-Consistency: The proposed method achieved an F1 score of 84.6% (Hierarchical-Consistency) and 74.6% (Majority-Consensus).
- Precision Gain: The method showed an 11.8% increase in precision over VLM baselines, indicating it is much better at rejecting false positives (incorrect trajectories).
Hyperparameters: The method is robust; performance stabilizes with as few as $N=10$ sampled trajectories and is insensitive to the clustering distance threshold $\tau$ over a wide range.

5. Significance and Limitations

Significance:

Training-Free Improvement: The method improves LLM performance without fine-tuning or additional training data, relying solely on sampling and geometric reasoning.
Bridging Text and Geometry: It provides a rigorous mathematical framework (Lie groups) to handle the semantic ambiguity of natural language prompts in visual generation.
Reliable Verification: It offers a more reliable alternative to VLMs for verifying geometric properties, which are often hallucinated by vision models.

Limitations:

Single Prototype Assumption: The current method assumes a shape family is defined by a single prototype and one warp. It struggles with prompts that map to multiple disjoint prototypes (e.g., "heptagram" could be {7/2} or {7/3}, or the letter "M" which has infinite variations).
LLM Distribution Assumptions: The decision criteria rely on assumptions about the LLM's output distribution (e.g., that correct answers are the majority or that incorrect answers don't cluster with correct ones). If the LLM consistently hallucinates a specific wrong shape, the method may converge on the wrong cluster.
Scope: Currently limited to 2D motion graphics trajectories; extension to 3D scenes or complex physics-based animations is future work.

In conclusion, this paper successfully adapts self-consistency to the visual domain by leveraging geometric invariance, offering a robust, unsupervised method to enhance both the generation and verification of LLM-driven motion graphics.