Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulation

This paper introduces "Talk to Your Slides," a high-efficiency slide editing agent that leverages language-driven structured data manipulation instead of visual perception to achieve faster, more accurate, and cost-effective text-centric and formatting modifications compared to Multimodal LLM-based GUI agents, supported by the newly proposed TSBench benchmark.

The Gist

Here is an explanation of the paper "Talk to Your Slides" using simple language and creative analogies.

The Big Problem: The "Pixel" vs. The "Blueprint"

Imagine you have a massive PowerPoint presentation with 50 slides, and your boss asks you to translate all the text from Korean to English, fix the spelling, and change the font color to blue.

• The Old Way (GUI Agents): Imagine hiring a robot that can only see the screen like a human does. It has to look at every slide, take a picture, read the text using OCR (like a scanner), figure out where the text is, click the mouse, type the new words, and check if it looks right.

  • The Analogy: This is like trying to fix a car engine by looking at a photo of the car and guessing where the bolts are. It's slow, prone to mistakes (the robot might miss a word), and very expensive because the robot has to "think" hard about every single pixel.

• The New Way (Talk-to-Your-Slides): Instead of looking at the picture, this new agent opens the blueprint of the presentation. It speaks directly to the software's internal code.

  • The Analogy: This is like having a master mechanic who opens the hood and talks directly to the engine's computer. They don't need to guess where the bolts are; they just send a command: "Change bolt #42 to blue." It's instant, precise, and cheap.

What is "Talk-to-Your-Slides"?

The researchers built an AI agent that acts as a translator between human language and computer code. Instead of trying to "see" the slides, it "reads" the structured data (the XML and object code) that makes up the PowerPoint file.

Here is how it works, broken down into four simple steps:

1. The Planner (Instruction Understanding):
   • Analogy: You tell a smart assistant, "Fix the slides." The assistant breaks this down into a checklist: "Slide 1: Translate text. Slide 2: Change font color." It creates a clear map of what needs to happen.
2. The Reader (Document Understanding):
   • Analogy: The assistant opens the "backstage" of the PowerPoint file. It doesn't look at the pretty pictures; it reads the raw data. It knows exactly which text box is where, what font is used, and what color it is. It turns this messy data into a neat, organized list (JSON).
3. The Editor (Document Editing):
   • Analogy: The assistant takes your checklist and the raw data list. It says, "Okay, for Slide 1, I need to change this specific text string to English and make it bold." It updates the list with the new information.
4. The Builder (Code Generator):
   • Analogy: Finally, the assistant writes a short computer script (Python code) that tells PowerPoint exactly how to apply those changes. It runs the script, and poof—the slides are fixed instantly.

Why is this a Big Deal?

The paper proves that for tasks involving text and formatting, looking at the screen is actually the wrong way to do it.

• Speed: It's 34% faster. Because it skips the "looking and guessing" part, it flies through 50 slides in minutes instead of hours.
• Accuracy: It's 34% more accurate. Since it reads the actual text code, it never misses a word or misreads a letter (a common problem for robots that just look at images).
• Cost: It's 87% cheaper. Processing images requires powerful, expensive AI models. Reading text code is much lighter and cheaper.

The "TSBench" Benchmark

The researchers realized there was no good test to see how well AI could edit slides (most tests only checked if AI could make slides from scratch). So, they built TSBench.

• The Analogy: Imagine a driving test. Before, we only tested if a car could drive in a straight line. TSBench is a complex obstacle course with 379 different challenges, like "Change the color of the third tire on the left" or "Translate the dashboard text." They even added a "Hard Mode" with tricky questions to see if the AI gets confused or gives up gracefully.

The Catch (Limitations)

The paper is honest about what it can't do yet.

• The Analogy: The agent is a master mechanic, but it's blind to the "vibe." If you say, "Make this slide look more balanced," the agent might struggle because "balanced" is a feeling, not a specific code instruction. It's great at moving text boxes and changing fonts, but it might need a human (or a visual AI) to check if the final design looks "pretty."

The Bottom Line

Talk-to-Your-Slides is a game-changer for office work. It stops treating PowerPoint like a picture you have to stare at and starts treating it like a database you can talk to. It turns a tedious, day-long chore into a few minutes of automated magic, saving time, money, and a lot of headaches.

In short: Stop looking at the screen; start reading the code.

Technical Summary

Here is a detailed technical summary of the paper "TALK TO YOUR SLIDES: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulation."

1. Problem Statement

Editing presentation slides (e.g., PowerPoint) is a frequent but tedious task involving creative layout design, text maintenance, and batch processing. Current solutions face significant limitations:

• Manual Editing: Time-consuming and prone to human error, especially for large-scale tasks like translating 50+ slides or updating pricing across multiple decks.
• GUI-Based Agents (Visual Approaches): Recent Multi-modal LLM (MLLM) agents (e.g., UFO) that control software via screen captures and mouse/keyboard interactions excel at spatial tasks but suffer from:
  • High Latency & Cost: Processing image inputs and simulating UI interactions is computationally expensive.
  • Fidelity Issues: Reliance on Optical Character Recognition (OCR) leads to text recognition errors and loss of formatting details.
  • Inefficiency: They struggle with text-centric or batch processing tasks where visual perception is unnecessary overhead.
• Direct Code Generation: Simple baselines that convert instructions directly to code often fail on complex tasks requiring contextual interpretation and multi-step planning.

Core Research Questions:

1. Can a non-visual, structure-aware agent achieve superior efficiency and fidelity compared to visual approaches for text-centric slide editing?
2. What system architecture enables the effective decomposition of complex instructions into precise executable steps?

2. Methodology: TALK-TO-YOUR-SLIDES

The authors propose TALK-TO-YOUR-SLIDES, an LLM-powered agent that operates by manipulating the underlying structured data (XML/COM object model) of slides rather than processing screen pixels. The system uses a hierarchical architecture with four distinct modules:

A. High-Level: Instruction Understanding

• Function: Interprets natural language user commands and decomposes them into structured, actionable plans.
• Mechanism: Uses an LLM (e.g., Gemini-1.5-flash) with in-context learning to parse instructions into a JSON list specifying target slides, elements (title, body, images), and specific actions.
• Output: A structured plan detailing what to modify and where, bridging high-level intent with low-level execution.

B. Low-Level: Document Understanding

• Function: Extracts a comprehensive, fine-grained representation of the slide's current state.
• Mechanism: Instead of converting slides to HTML (which loses metadata), the system uses a custom rule-based parser via COM (Component Object Model) on Windows (or AppleScript on macOS).
• Granularity: It parses at the run-level (contiguous text segments with consistent formatting) rather than just the placeholder level. This captures precise font styles, colors, and positions for every text run, shape, and image.
• Output: A structured JSON representation of the slide's Document Object Model (DOM).

C. High-Level: Document Editing

• Function: Modifies the parsed JSON data based on the plan generated in Step A.
• Mechanism: An LLM (e.g., Gemini-2.5-flash) takes the original JSON and the editing plan to generate the "edited" JSON. It ensures that only specified elements are changed while preserving the formatting of untouched elements.

D. Low-Level: Code Generator

• Function: Translates the difference between the original and edited JSON into executable code.
• Mechanism: Generates Python code (using win32com for Windows) that directly interacts with the PowerPoint application instance to apply changes.
• Self-Reflection: The system includes a feedback loop. If code execution fails (e.g., type errors), the error message is fed back to the LLM to regenerate the code, iterating until success or a max attempt limit is reached.

3. Key Contributions

1. Language-Driven Structural Agent: Demonstrates that for text-centric and batch editing, manipulating structured data (XML/COM) is significantly more efficient and accurate than visual GUI agents. It avoids the overhead of image processing and OCR.
2. Hierarchical Framework: Introduces a novel architecture that separates high-level reasoning (planning/editing) from low-level execution (code generation), effectively bridging the gap between user intent and application objects.
3. TSBench Benchmark:
   • A new, human-verified benchmark dataset containing 379 diverse editing instructions across four categories: Text Editing, Visual Formatting, Layout Adjustment, and Slide Structure.
   • TSBench-Hard: A challenging subset designed to test robustness against visually dependent, ambiguous, and multi-step tasks (e.g., "Align text box to the left edge of the image").
4. Comprehensive Evaluation: Provides a rigorous comparison against state-of-the-art baselines (UFO GUI agent and Direct Code Generation) across performance, efficiency, and cost.

4. Experimental Results

The system was evaluated using the TSBench dataset with various LLM backbones (Gemini-2.5-flash, DeepSeek V3, GPT-4.1-mini).

• Performance (Success Rate & Fidelity):
  • Success Rate (SR): TALK-TO-YOUR-SLIDES achieved an overall SR of 96.84% (with DeepSeek) and 96.83% (with Gemini), significantly outperforming the GUI-based UFO agent (75%) and Direct Code Generation (60-77%).
  • Instruction Fidelity: The method achieved 34% better instruction fidelity compared to GUI baselines, particularly in text editing and formatting tasks where OCR failures plague visual agents.
• Efficiency (Speed & Cost):
  • Speed: Processing was 34% faster (average execution time reduced from ~114s to ~71s).
  • Cost: The method operated at an 87% lower cost (approx. $0.0014 vs $0.0055 per instruction) because it avoids expensive image token processing and reduces the number of API calls required for visual verification.
• Robustness (TSBench-Hard):
  • On the difficult "Hard" subset, the proposed method achieved a 31.3% success rate, outperforming the best baseline (UI Agent) by +16.5%.
  • It demonstrated superior Refusal Accuracy (64.7%) on impossible tasks (e.g., identifying audio speakers in a video), correctly identifying when a task was beyond its capabilities rather than hallucinating.

5. Significance and Future Directions

• Paradigm Shift: The paper argues that for structured document editing, direct access to the object model is superior to visual perception. It proves that "seeing" the screen is often unnecessary and detrimental for tasks requiring precise text manipulation.
• Scalability: The approach enables real-time, large-scale batch editing (e.g., translating an 89-slide lecture deck in minutes for under $10), making it viable for enterprise and academic workflows.
• Limitations & Hybrid Future: The authors acknowledge that purely code-based approaches struggle with purely perceptual tasks (e.g., "make this look balanced") or complex spatial reasoning where visual feedback is needed. They propose a Hybrid Agent future where the efficient structural manipulation handles the bulk of batch tasks, while VLM capabilities are selectively invoked for final visual quality assurance and complex layout adjustments.

In conclusion, TALK-TO-YOUR-SLIDES establishes a new standard for software automation in presentation editing, prioritizing structural precision and cost-efficiency over visual simulation.