💬 NLP

fog: Expressing Motion and Emotion through Function Composition of AI-Generated Code

This paper introduces "fog," a function composition framework that leverages AI-generated code and an interactive animation editor to enable users to create expressive, Heider-Simmel-style motion and emotion, which was validated through perceptual studies showing 68% semantic recognition accuracy and improved user control.

Original authors: Vivian Liu, Lydia Chilton

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vivian Liu, Lydia Chilton

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to tell a story using nothing but floating shapes on a screen—a triangle, a circle, a square. You want the circle to look angry as it chases the triangle, or the triangle to look scared as it runs away. Sounds simple, right? But ask an AI to "make the circle look angry," and you might get a circle that just spins in place or moves too fast, missing the feeling entirely.

That's the problem fog (a new tool by researchers Vivian Liu and Lydia Chilton) is trying to solve. Instead of just shouting vague instructions at a computer, fog treats motion and emotion like a giant, mix-and-match LEGO set built from code.

The Big Idea: Motion is a Recipe, Not a Magic Spell

The researchers suggest that motion and emotion aren't just random chaos; they are made of specific ingredients. Think of it like cooking. If you want a spicy dish, you don't just tell the chef, "Make it spicy." You add specific amounts of chili, pepper, and heat.

In fog, the "ingredients" are Verbs (like chase or flee), Adverbs (like hesitantly or aggressively), Gestures (like waving or shaking), and Emotions (like anger or fear). The magic happens when you snap these code-blocks together.

For example, you can tell the AI to make a shape hesitantly(chase) another shape. The system doesn't just guess; it builds a specific recipe where the shape moves forward but stops and starts nervously, maybe shaking a little. Or, you could combine anger with a lunge, creating a motion that builds up energy, strikes hard, and then recoils.

The "Magic" Test: Can People Actually See the Feeling?

The team didn't just build this and hope for the best. They ran a massive taste test with 452 different animations. They showed these moving shapes to hundreds of people and asked, "What is this shape doing? Is it chasing, avoiding, or fighting?"

The results were pretty cool. People were able to guess the correct meaning 68% of the time. That might sound like a C+ in school, but remember, if they were just guessing randomly among four options, they would only get it right 25% of the time. So, fog's animations were 2.68 times better than a lucky guess.

However, the paper is careful to point out that this isn't a perfect solution. When they tried to mix two things together—like making a shape hesitantly and chase—the accuracy dropped a bit (to about 58%). Sometimes the "ingredients" clashed, like trying to mix oil and water. The researchers found that if the code for "hesitant" and the code for "chase" tried to control the same part of the movement, they sometimes canceled each other out. It's a reminder that while the system is powerful, it's not yet a mind-reader.

The "Playground" for Creators

The researchers also built a playground (an animation editor) to see how real people would use this. They invited 10 people—some who know animation like the back of their hand, and some who have never coded a thing.

They compared fog to a standard "just type what you want" AI tool. The difference was huge.

The Old Way: Users had to type a prompt, wait for the AI to generate a whole new scene, and then wait again if they didn't like it. It took about 1.75 minutes just to see a new version of the animation.
The fog Way: Users could tweak the motion instantly. They could drag a slider to make a shape move faster, draw a path for it to follow, or mix and match emotions on the fly. The time to see a new version dropped to just 0.36 minutes.

The pros loved the control, saying it felt like having "atomic" building blocks to fine-tune every detail. The beginners loved the safety net; instead of staring at a blank screen wondering what to type, they could pick from a grid of pre-made "verbs" and "adverbs" to get started. One beginner even managed to animate a circle that looked anxious and then calmed down, saying, "I can see the emotion in the circles."

What It's NOT (And What It Can't Do Yet)

It's important to know what this paper doesn't claim.

It's not a movie maker: The animations are still just simple shapes (circles and triangles) moving around. The paper explicitly says that while 68% accuracy is great, you can't expect 100% perfection. A simple circle just can't express every possible human emotion or complex story.
It's not a "fix-all" for AI: The researchers tried to have the AI "self-refine" its own code to fix the clashing ingredients, but that didn't really help much. The accuracy barely moved.
It's not for complex robots (yet): The system works best for these abstract shape stories. The paper notes that if you want to animate a real robot with joints and lighting, or a character with a face, fog isn't ready for that level of detail just yet.

The Takeaway

fog suggests that we can teach computers to understand emotion by breaking it down into code functions, like a recipe book for feelings. It's not a perfect, magic wand that solves every animation problem, but it's a powerful new way to play. It turns the scary, vague process of "telling an AI what to do" into a fun, hands-on game of mixing and matching motion ingredients, letting both experts and beginners tell stories with shapes that actually feel alive.

Technical Summary: fog – Expressing Motion and Emotion through Function Composition of AI-Generated Code

Problem Statement
Animating motion and emotion, particularly in the context of Heider-Simmel style abstract storytelling, presents significant challenges in control and iteration. While AI can generate visual content, users struggle to precisely define dynamic motion attributes such as speed, force, easing, and timing across space and time. Existing AI tools often lack mechanisms for users to easily explore, iterate, and control these motion dynamics. Furthermore, traditional animation frameworks often focus on physics or character rigging rather than the semantic and emotional expressiveness required to convey narrative intent through simple shapes. The core problem addressed is bridging the symbolic nature of storytelling (verbs, emotions) with the numerical underpinnings of motion (velocity, acceleration, energy) to provide users with granular control over AI-generated animation.

Methodology
The paper introduces fog, a function composition framework that utilizes Large Language Models (LLMs) to generate code for motion and emotion. The system is built upon a hierarchy of abstract classes derived from object-oriented programming principles and theories from computer graphics and cognitive psychology.

Abstract Class Hierarchy: The framework defines a structured vocabulary comprising:
- Scenes and States: The highest-level representation, where States describe the animation of Entities.
- Entities: Represented as shapes (circles, triangles) with properties like position, velocity, energy, and orientation. They utilize squash-and-stretch and motion trails to simulate animacy.
- Motion Classes:
  - PrimaryMotion: Represents the core trajectory or action (e.g., chase, flee, walk).
  - SecondaryMotion: Layers additional motion on top of the primary action (e.g., hesitantly, aggressively, wave).
- Semantic Categories: The hierarchy includes Verbs (Solo, Relational, PathMotion, PinMotion), Adverbs (modifiers like slowly, urgently), Gestures (small semantic units like nod, clap), and Emotions (affective states like anger, fear composed of discrete phases).
Execution Model: The system enforces a separation where PrimaryMotion updates the entity's trajectory, while SecondaryMotion (Adverbs, Gestures, Emotions) decorates or modifies this trajectory. Functions can compose additively or commutatively, writing to shared channels like velocity, acceleration, and energy.
AI Implementation: LLMs (specifically Claude-Opus-4-6) are prompted with the abstract class definitions and minimal guidelines to generate the specific code implementations for these functions.
User Interface: fog is implemented as an animation editor featuring a Main Canvas, Timeline, and State Editor. It supports:
- On-the-fly Generation: Users can generate new functions via prompts or AI suggestions.
- Direct Manipulation: Users can draw paths and adjust parameters via dynamically generated UI elements (e.g., 2D minimaps for speed and intensity).
- Function Composition: A visual grid allows users to combine Verbs, Adverbs, Gestures, and Emotions.

Key Contributions

fog Framework: A function composition system that expresses motion and emotion through AI-generated code, introducing a hierarchy of abstract classes (Verbs, Adverbs, Gestures, Emotions) to create an open-ended motion vocabulary.
Animation Editor Interface: A demonstration of how users can generate functions dynamically, define compositions interactively, and utilize both prompts and direct manipulation for editing.
Perceptual Evaluation: A study involving 452 fog-generated animations to test the recognizability of semantic motion meanings.
Mixed-Methods User Study: A study with 6 professionals and 4 novices evaluating fog's ability to support rapid iteration, exploration, and control compared to a prompt-only baseline.

Results

Perceptual Recognition: In a 4-alternative forced-choice (4AFC) task, users recognized the semantic meaning of fog-generated motion with 67.8% accuracy (approximately 2.68x improvement over a 25% chance baseline).
- Verbs: 77.7% accuracy.
- Gestures: 78.0% accuracy.
- Adverbs: 62.5% accuracy.
- Emotions: 64.2% accuracy.
- Compositions: Accuracy varied, with Verb + Gesture at 69.8%, Adverb + Verb at 58.3%, and Emotion + Verb at 57.8%. The study noted that misalignment in "channels" (e.g., an adverb targeting acceleration while a verb targets speed) could reduce compositional accuracy.
User Study Findings:
- Efficiency: Users in the fog condition achieved a significantly faster feedback cycle, with an average of 0.36 minutes between new animations compared to 1.75 minutes in the prompt-only baseline ( $p \le 0.001$ ).
- Control and Iteration: Participants reported that fog offered better control over output and allowed for more granular, atomic editing. Professionals appreciated the ability to create bespoke behaviors and blend emotions, while novices found the presets and motion grid provided necessary scaffolding to overcome the difficulty of writing effective prompts.
- Limitations: Some participants noted that the system felt like "programming" and that the minimap exploration sometimes overpromised subtle changes.

Significance and Claims
The paper claims that fog demonstrates a viable approach to bridging symbolic storytelling with numerical motion control through code generation. By treating motion and emotion as composable functions within an abstract class hierarchy, the system enables users to:

Express complex dynamics: Create open-ended vocabularies for motion that go beyond simple prompts.
Iterate rapidly: Significantly reduce the time between idea generation and visual feedback compared to pure prompt-based workflows.
Maintain control: Provide a structured surface (function composition) that mitigates the "black box" nature of generative AI, allowing users to refine specific aspects of motion (speed, energy, path) without rewriting entire scripts.

The authors conclude that abstract class definitions can serve as a new "surface area" for AI prompts, offering a middle ground between rigid traditional UIs and unstructured natural language generation. They acknowledge limitations, noting that Heider-Simmel animations cannot express all possible human nuances and that the current channel-based representation may struggle with highly similar emotions or adverbs. Future work is suggested to include event-based triggers and more complex code structures to support richer social dynamics.