Technical Summary: Spatula – Exploring On-Demand In-Situ Interfaces and Interaction for Attribute Control

1. Problem Statement

While generative AI has lowered the barrier to content creation, refining generated results remains a significant challenge. Current workflows suffer from a disconnect between high-level intent and low-level parameter control:

• LLM-based Workflows: Rely on text prompts for refinement, which are indirect, lack immediate feedback loops, and offer poor predictability regarding the magnitude of change or specific elements affected.
• Traditional Authoring Tools: Expose rich parameter spaces through hierarchical panels, enabling precision but imposing high cognitive loads and disrupting iterative flow due to UI clutter.
• Existing On-Demand Interfaces: While some systems generate controls dynamically, they often treat them as external, ad-hoc additions (e.g., disconnected side panels) rather than integrated parts of the creative process. They frequently fail to provide the visual scaffolding needed to reason about high-dimensional, interdependent parameters.

The core challenge identified is not merely generating interfaces, but structuring control to make it actionable, intuitive, and integrated into the visual canvas.

2. Methodology

2.1 Formative Studies

The authors conducted a two-stage formative study to ground the design:

1. Interviews & Design Elicitation: Six experienced motion graphics designers were interviewed to compare traditional and LLM-based workflows. Key limitations identified included excessive low-level control in traditional tools and obscured parameters in LLM workflows.
2. Technical Probe: A prototype system was developed where an LLM analyzes motion graphics code (p5.js), identifies adjustable attributes, and generates on-demand, in-situ UI widgets (e.g., drag handles, sliders) directly on the canvas.
   • Findings: Users appreciated the shift to interactive editing but reported four key gaps: (C1) Limited discoverability of controls, (C2) Insufficient control resolution (lack of multi-level precision), (C3) Limited support for collective editing of related elements, and (C4) Incomplete coverage of potential controls.

2.2 System Design: The Elastic Attribute Control Space

Based on the formative findings, the authors propose Spatula, a system that operationalizes an Elastic Attribute Control Space. This space is adaptive and continuously reconfigurable, structured along four dimensions:

• Discoverability (In-Situ Space Discovery):
  • Mechanism: Context-aware, progressive hints triggered by user intent (hover, focus, manipulation).
  • Implementation: The system highlights interactable elements, provides localized interaction descriptors (e.g., "Drag to change speed"), and reveals real-time parameter values during manipulation.
• Resolution (Multi-Resolution Control):
  • Mechanism: Elastic widgets that expand from coarse to fine-grained controls upon user invocation.
  • Implementation: Users start with low-resolution controls (e.g., a scalar slider) for rapid exploration. When higher precision is needed, the interface elastically expands in-situ to richer representations (e.g., Bézier curve editors, continuous HSV color spaces).
• Scope (Semantic Scope Control):
  • Mechanism: Collective adjustment of semantically or visually related elements.
  • Implementation: The system identifies groups of elements based on role, appearance, or function. Users can toggle "Group Mode" to manipulate attributes across the entire set simultaneously, while retaining the ability to switch to individual editing.
• Expandability (Proactive Space Expansion):
  • Mechanism: Dynamic extension of the control space when existing parameters are insufficient.
  • Implementation: The system proactively suggests a set of potential attributes. If these are insufficient, users can provide lightweight textual input to describe missing controls, prompting the system to generate new UI elements on demand.

2.3 Implementation

• Architecture: A web-based environment using React.js (frontend) and Node.js (backend), centered on an HTML5 Canvas rendered via p5.js.
• LLM Agent: Utilizes Gemini-3-Pro to analyze code context, infer attributes, and synthesize interaction logic.
• Pipeline:
  1. Parse: The system analyzes p5.js code to extract editable attributes and infer interaction mappings.
  2. Synthesize: An "Applying Agent" selects UI primitives from a knowledge base and injects them as a temporary interactive overlay into the running animation.
  3. Interact: Users manipulate elements directly on the canvas. Shortcuts (e.g., 'h' for highlighting, 'j' for hints) manage the interface state.
  4. Export: Finalized parameters are mapped back to the original code, removing temporary scaffolds for a clean export.

3. Key Contributions

1. Reframing Attribute Control: The paper identifies the lack of structured control as a key limitation in generative workflows and reframes attribute control as an explorable, structured, and elastic space rather than a static set of UI elements.
2. Elastic Attribute Control Space: The authors derive and define four mechanisms (Discoverability, Resolution, Scope, Expandability) for constructing and interacting with this space.
3. Spatula System: A proof-of-concept, on-demand, in-situ interface that enables real-time, fine-grained parameter manipulation via context-aware control scaffolds.
4. Empirical Insights: A comparative user study (N=12) providing insights into how users of varying expertise (novices vs. professionals) explore and utilize attribute control.

4. Results

A comparative user study involving 12 participants (6 novices, 6 professionals) compared Spatula against four conditions: integrated LLM text-to-motion, separate panel adjustment, the initial tech probe, and the full Spatula system.

• Enhanced Engagement: Participants reported that low-level manipulation via in-situ controls enhanced aesthetic engagement and creative exploration compared to high-level text prompts. Users felt more in control and were more likely to fine-tune results to match personal vision.
• Divergent Interaction Paths:
  • Novices: Adopted a bottom-up, exploratory approach. They used in-situ hints as "pedagogical sandboxes" to understand animation principles through immediate visual feedback (e.g., dragging curves to see motion effects).
  • Experts: Adopted a top-down, sculpting approach. They leveraged features like semantic scope control to efficiently block out scene-level changes and used proactive expansion to surgically add missing parameters.
• Complementary Benefits: While 11/12 participants praised the intuitiveness of in-situ interaction, some noted that separate panels were useful when in-situ controls occluded the visual workspace. This suggests a need for adaptive interface generation that can provide both modes.

5. Significance and Claims

The paper claims that Spatula represents a shift toward Generative Interaction, where the user interface is not a static artifact but a transient, functional scaffold synthesized by an agent in response to user intent.

• Interaction Compilers: The authors posit that LLMs can act as "interaction compilers," translating high-level semantic intent into functional, in-situ code snippets. This bridges the gap between the "magic" of zero-shot generation and the "control" of manual manipulation.
• Modularity: The design is presented as a modular, plug-and-play component. While the study focused on motion graphics, the authors demonstrate its applicability to web design (CSS property manipulation) and 3D modeling (geometry transformation), suggesting the paradigm generalizes to other creative domains.
• Limitations: The authors modestly acknowledge limitations, including the potential for overwhelming attribute spaces in complex contexts, the difficulty of adaptive intent prediction, and the ambiguity between "attributes" and fundamental content restructuring (e.g., hard-coded vs. procedural shapes). They conclude that Spatula serves as a complementary approach to existing workflows rather than a universal replacement.