An Embodied Companion for Visual Storytelling

This paper introduces "Companion," an embodied AI system that integrates a drawing robot with Large Language Models to facilitate bidirectional, speech-and-sketch-based co-creation, transforming the robot from a passive tool into a playful artistic partner capable of generating professionally recognized visual stories.

The Gist

Imagine you are sitting at a table with a friend, a pen in your hand, and a blank piece of paper between you. You start drawing a simple circle. Your friend looks at it, smiles, and adds a few lines to turn it into a sun. Then, you add a cloud, and they add a bird. You aren't just taking turns; you are building a story together, reacting to each other's moves in real-time.

Now, imagine that your friend is a robot. But not a robot that just follows a strict list of instructions like a factory machine. This robot is a playful, chatty partner that can listen to your voice, look at your drawing, and decide to add its own twist to the story.

This is the core idea behind "Companion," a new project by artist Patrick Tresset and Google DeepMind.

The Problem: The "Ghost" in the Machine

For years, artists have used robots to draw. Usually, the process works like this: The artist programs the robot with a set of rules, steps back, and lets the machine do the work. The robot draws perfectly, but the artist's "hand" (their immediate presence and spontaneous decisions) is missing from the actual drawing process. It's like writing a letter, sealing it, and mailing it to a stranger who then writes the reply. You never actually talk to them.

The creators of Companion felt something was missing. They wanted the robot to be a co-pilot, not just an autopilot. They wanted the artist and the machine to be in the same room, making decisions together, right then and there.

The Solution: A Robot with a Brain and a Voice

To build this, they combined three things:

1. A Physical Robot Arm: A low-cost, slightly wobbly robot that actually holds a pen and draws on paper. Its slight imperfections (like a shaky hand) give the art a unique, human-like feel.
2. A "Brain" (Large Language Model): They connected the robot to a powerful AI (like the ones behind Google's Gemini). This AI understands language, stories, and art history.
3. A "Vocabulary" of Drawing: They taught the AI how to draw by showing it examples of how to make specific shapes (like a tree or a face) using the robot's specific tools. This is called In-Context Learning. Think of it as showing the robot a picture book of "how to draw" before you start the game.

How It Works: The Dance of Creation

The interaction is like a lively dance:

• You speak: You tell the robot, "Let's draw a story about a bird losing a feather."
• You draw: You might draw a quick circle on the paper.
• The Robot reacts: The robot looks at your circle, listens to your story, and decides, "Oh, that circle is a bird! I'll add a wing." It might even say out loud, "I think this bird is scared!" (It speaks with a funny, non-native accent to sound more like a unique character, not a boring computer).
• The Robot draws: It moves its arm to add the wing.
• You react again: You see the wing and decide to add a cloud.

Sometimes, the robot surprises you. In one experiment, an artist drew a car high up on the page. Instead of just drawing a road, the robot interpreted the car as floating and added a character floating on a cloud next to it. It took the idea and ran with it in a direction the human hadn't expected.

The "Magic" of the Result

The researchers tested this system with art experts. They wanted to know: Is this just a weird machine, or is it real art?

The experts said yes, it's art.

• It has a style: The drawings look like a mix of ancient cave paintings, child-like sketches, and modern art. They aren't perfect, but that's the point. The "wobbly" lines tell a story of the robot's physical movement.
• It tells stories: The robot could illustrate fables (like "The Tortoise and the Hare") or song lyrics, showing the passage of time by drawing the same character in different spots on the page.
• It feels like a partnership: The experts felt that the robot wasn't just a tool being used; it was a partner with its own "personality." One expert noted that the robot's hesitant lines made it look like it was "learning to see" just like a human.

Why This Matters

This project changes how we think about AI in art. Instead of AI being a magic box that spits out a finished image (like a text-to-image generator), Companion is about the journey.

It's the difference between ordering a pizza (you get a result) and cooking a meal together with a friend (you share the experience). The robot isn't replacing the artist; it's inviting the artist into a new, unpredictable adventure where the final picture is a surprise for both of them.

In short: Companion is a robot that doesn't just draw for you; it draws with you, turning a quiet room into a shared playground of stories and sketches.

Technical Summary

Here is a detailed technical summary of the paper "An Embodied Companion for Visual Storytelling."

1. Problem Statement

The paper addresses a critical gap in the history of robotic art: the transition from autonomous execution to synergistic co-creation.

• The Limitation of Autonomy: Previous robotic art systems (including the authors' earlier work) focused on total machine autonomy, where the robot draws independently based on pre-programmed intent. While consistent, these works often lacked the "meaning and feeling" derived from real-time human experience and direct presence.
• The "Black Box" of Digital AI: Purely digital AI (e.g., text-to-image models) lacks the temporal, physical process of mark-making. They generate static images rather than unfolding narratives in time and space, and they often adhere to known aesthetic patterns without the "surprise" of physical interaction.
• The Goal: To reintroduce the artist's hand and presence into the execution process, shifting the robot from a passive tool to an embodied, playful co-creative partner capable of driving shared visual storytelling into unexpected aesthetic territories.

2. Methodology

The authors developed Companion, a prototype system integrating a physical drawing robot with Large Language Models (LLMs). The methodology relies on a closed-loop interaction between human, machine, and physical environment.

A. System Architecture

• Hardware (Embodiment):
  • A custom-built, low-cost planar 4-joint robotic arm (shoulder, elbow, wrist, pen-lift) using Dynamixel servos.
  • Workspace: 170 × 130 mm.
  • Aesthetic Design: Unlike industrial arms, the robot's mechanical imperfections (compliance, backlash, frame flexibility) are intentionally utilized as aesthetic features to create variable line quality.
  • Perception: A Logitech C920 camera (pan-tilt) captures the drawing surface. Images undergo perspective rectification and Contrast Limited Adaptive Histogram Equalization (CLAHE) to detect faint human marks.
  • Interaction: Primarily speech-driven (microphone) with non-verbal physical cues (rotating joints to signal turn-taking).
• Software (The "Brain"):
  • Core Agent: A Python-based agent interfacing with the Gemini API.
  • Control Framework: YARP (Yet Another Robot Platform) for low-level control, chosen for its lightweight architecture over ROS.
  • Toolset: The LLM utilizes function calling to execute specific drawing primitives:
    • Primitives: Lines, segments, circles, text (vector Hershey fonts).
    • Procedural: Scribbles, hatching (using rejection sampling or geometric intersection).
    • Organic: Splines and paths (cubic interpolation) for fluid motion.
    • Composite: "Scribbly-splines" to mimic sketching styles.

B. Core Technical Innovations

1. In-Context Learning (ICL) for Visual Vocabulary:
   • Instead of fine-tuning the model, the system uses ICL to teach the LLM a "graphic vocabulary."
   • The system provides the LLM with pairs of visual examples (drawings) and procedural descriptions (how to draw them using specific tools).
   • This allows the agent to infer stylistic rules (e.g., "use splines for organic forms," "use hatching for ground") and apply them to new subjects without parameter updates.
2. System Instructions & Persona:
   • The LLM is prompted with a specific persona: an "imaginative, story-telling drawing robot" inspired by cave paintings and children's art.
   • Constraints are enforced via prompts (e.g., stick figures for people, 40-pixel safety buffers, specific line-weight simulation via retracing).
3. Bidirectional Interaction Loop:
   • Input: Human speaks or physically rotates the robot's joints.
   • Perception: Camera captures the current state (human marks + robot marks).
   • Reasoning: The LLM processes the audio and visual context to decide on the next action (draw, speak, or wait).
   • Output: The LLM generates conversational text (synthesized with a "non-native" accent for character) and tool calls that drive the robot.

3. Key Contributions

• Shift from Delegation to Collaboration: The paper demonstrates a paradigm shift where the AI is not just executing commands but acting as an agent with "emergent agency," capable of rejecting suggestions, interpreting abstract prompts, and driving the narrative in unexpected directions.
• Procedural Style Transfer via ICL: The authors show that pairing visual examples with procedural instructions allows LLMs to overcome the "translation gap" between seeing an image and understanding the motor primitives required to draw it. This results in a consistent, emergent aesthetic style.
• Embodied Visual Storytelling: The system successfully translates narrative concepts (fables, song lyrics, abstract sentences) into sequential physical drawings, utilizing strategies like spatial repetition (time passing) and symbolic modification (elongated limbs for action).
• Physicality as Aesthetic: The system leverages the mechanical "noise" of the robot (friction, backlash) as a deliberate stylistic element, creating a "memory of computational and physical forces" rather than a clean digital replication.

4. Results

The system was evaluated through a series of experiments and a formal expert assessment:

• Experimental Roles:
  • Artistic Director: The human guided the story; the robot interpreted instructions, sometimes deviating creatively (e.g., drawing a "long arm" to catch a feather instead of moving the character).
  • Autonomous Illustrator: The robot illustrated fables (e.g., "The Hare and the Turtle") and abstract texts (Beckett's "Fail better") without human intervention, successfully depicting time and sequence.
  • Co-Creator: In shared sessions, the robot and artist took turns, building a narrative together (e.g., a "floating car" story).
• Model Evolution: Experiments across different Gemini model versions (1.5 to 2.5) showed a clear progression in legibility and complexity, with newer models producing more recognizable subjects and better adherence to narrative constraints.
• Consensual Assessment Technique (CAT):
  • Panel: 7 art-world experts (curators, collectors, artists).
  • Metrics: Evaluated on Aesthetic Identity, Global Quality, Originality, and Conscious Intent.
  • Scores:
    • Aesthetic Identity: High (M=6.0/7). Experts noted a distinct, cohesive style comparable to "Pre-historic petroglyphs," "Paul Klee," and "Basquiat."
    • Global Quality & Originality: High (M=5.71 and 5.86), validating the work for professional exhibition.
    • Conscious Intent: Lower (M=4.71), reflecting the experts' discernment regarding machine agency, though they acknowledged a "shared authorship."
  • Outcome: The system produced works with a "child-like" or "naive" quality that experts viewed as a strength, allowing audiences to project their own stories.

5. Significance

• Artistic Validation: The system has moved beyond experimental prototypes to produce works accepted for professional publication (a book of 24 drawings) and solo exhibitions.
• Theoretical Impact: It challenges the notion of AI as a mere tool, proposing a model of embodied companionship where the physical presence of the machine and the cognitive capabilities of the LLM create a unique "third space" of creativity.
• Future Directions: The research opens pathways for:
  • Observational drawing by AI.
  • Self-improving tool development by the agent.
  • Assistive technologies for individuals with motor disabilities to draw through collaborative AI.

In summary, Companion successfully demonstrates that by combining embodied robotics with LLM-driven in-context learning, it is possible to create an AI partner that does not just execute art but collaborates in the process of visual storytelling, generating a unique, emergent aesthetic identity.