💻 computer science

Auditing and Controlling AI Agent Actions in Spreadsheets

This paper introduces Pista, a spreadsheet AI agent designed to decompose execution into auditable, controllable actions, demonstrating through user studies that active human participation in decision-making—not just post-hoc review—is essential for effective oversight, error detection, and a sense of co-ownership in AI-driven knowledge work.

Original authors: Sadra Sabouri, Zeinabsadat Saghi, Run Huang, Sujay Maladi, Esmeralda Eufracio, Sumit Gulwani, Souti Chattopadhyay

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sadra Sabouri, Zeinabsadat Saghi, Run Huang, Sujay Maladi, Esmeralda Eufracio, Sumit Gulwani, Souti Chattopadhyay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a super-smart, lightning-fast assistant to prepare your taxes. You give them the instructions, and they go into your spreadsheet, crunching numbers, writing formulas, and organizing data.

The Problem with Current AI:
Right now, most AI agents work like a black box. You hand them the ingredients, and they disappear into a kitchen. When they come back, they hand you a finished meal. You have to taste it and hope it's good. If the soup is too salty, you can't tell which ingredient they added wrong, or why they decided to use salt instead of pepper. You didn't see them cook, so you can't stop them from making a mistake until the meal is already ruined.

The Solution: Pista
The paper introduces Pista (which means "path" or "trail" in Italian/Spanish). Pista is a new kind of AI assistant for spreadsheets that changes the game from "black box" to "glass kitchen."

Instead of disappearing and coming back with a result, Pista cooks step-by-step, letting you watch and help along the way.

Here is how Pista works, using simple analogies:

1. The "Trail of Crumbs" (Traceability)

Imagine you are walking through a forest. A normal AI just teleports you to the destination. Pista leaves a trail of breadcrumbs.

How it works: Pista breaks the big task into small, manageable steps. Before it does the next thing, it pauses and says, "I'm about to add this formula to calculate the profit. Here is exactly what I'm doing and why."
Why it helps: You aren't just seeing the final number; you see the logic behind it. It's like seeing the chef say, "I'm adding two cups of flour because the recipe calls for a thick batter," rather than just seeing a bowl of dough appear.

2. The "Magic Eraser" (Localized Editing)

In a normal spreadsheet, if the AI makes a mistake in step 3, you might have to delete the whole thing and start over.

How it works: With Pista, if you see a step you don't like, you can say, "Wait, use 3 decimal places instead of 2," and Pista fixes only that specific step. It doesn't break the rest of the work.
The Analogy: It's like editing a sentence in a story. You don't have to rewrite the whole book just because you want to change one adjective. You just change the word, and the story continues.

3. The "Choose Your Own Adventure" (Branching)

This is the coolest feature. Sometimes you aren't sure if the AI's plan is the best plan.

How it works: If you want to try a different approach, Pista doesn't delete your current work. Instead, it creates a branch. It's like a tree. You stay on the main trunk, but you can step onto a side branch to see what happens if you calculate taxes differently.
The Analogy: Imagine you are driving. The GPS (the AI) suggests a route. You think, "Maybe we should take the scenic route instead." With Pista, you don't have to cancel the trip. You just click a button to create a "scenic route" version of the trip. You can drive both routes in your mind, compare them, and then decide which one to keep.

4. The "Scaffolding" (Guidance)

Sometimes you don't know what to ask the AI.

How it works: Pista acts like a helpful tour guide. It doesn't just wait for you to speak; it asks, "Hey, did you mean to handle missing numbers this way?" or "Here are three different ways we could calculate this."
The Analogy: It's like a GPS that says, "You're approaching a fork in the road. Do you want to go left (fastest) or right (scenic)?" It helps you think of options you might have forgotten.

What Did the Study Find?

The researchers tested Pista with real people doing financial tasks. Here is what happened:

Trust: People trusted Pista more, not because the answers were perfect, but because they saw the work being done. It's the difference between trusting a friend who shows you their homework vs. trusting a friend who just hands you the answer sheet.
Fewer Mistakes: People caught errors while the AI was working, rather than finding them after the fact.
Less Stress: People felt like co-authors of the spreadsheet, not just passive receivers. They felt in control.
Efficiency: Even though they were watching every step, they actually finished the tasks faster and with fewer confusing instructions because they could make small, precise fixes instead of restarting the whole thing.

The Big Takeaway

The main lesson of this paper is that we shouldn't just be reviewers of AI; we should be participants.

When AI does complex work, we need to be in the car with the driver, not just waiting at the destination. By letting us see the "trail," fix small things instantly, and try out different paths without fear, Pista turns AI from a mysterious magic trick into a reliable, collaborative partner.

1. Problem Statement

The paper addresses a critical gap in the deployment of AI agents for knowledge work, specifically within spreadsheet environments. While AI agents can autonomously perform complex, multi-step tasks, current systems operate as "black boxes" where the user is absent during execution.

The Core Issue: In spreadsheets, the artifact (the file) and the computation (the logic/formulas) are inseparable. When an agent acts, it directly modifies the user's cells.
Limitations of Current Systems: Existing agents (e.g., SheetCopilot, SheetMind) execute tasks in a single pass and return a final result. This forces users to perform post-hoc review, where they must reconstruct the agent's reasoning from the final output.
Consequences: This lack of transparency leads to:
- Inability to detect errors before they propagate.
- Difficulty in verifying assumptions or intent alignment.
- High cognitive load for users trying to reverse-engineer logic.
- A lack of user agency, resulting in passive acceptance of potentially flawed outputs.

2. Methodology

The authors employed a mixed-methods research approach involving the design, implementation, and evaluation of a new system called Pista.

A. System Design: Pista

Pista is a Microsoft Excel add-in that fundamentally changes the interaction model from "execute-and-return" to "trace-and-steer."

Decomposition: Instead of a single execution, Pista breaks the agent's plan into discrete, traceable steps.
Key Features:
1. In-Situ Explanations: At each step, the agent provides a natural language explanation of the action, the underlying formula, and the data range affected.
2. Localized Editing: Users can issue corrections scoped to the current step without restarting the entire task.
3. Branching Exploration: If an edit alters the logic for downstream steps, Pista automatically creates a new execution branch. Users can toggle between branches to compare outcomes.
4. Scaffolding: The system suggests follow-up questions and candidate edits to help users formulate intent and identify potential issues ("unknown unknowns").
Implementation: Built as a web/desktop Excel add-in using JavaScript/HTML (frontend) and Python Flask (backend). It utilizes the Gemini LLM (3.0 Flash and 3.1 Flash-Lite) for code generation and reasoning, structured via DSPy programs.

B. Evaluation Studies

Formative Study ( $N=8$ ):
- Goal: Identify challenges with baseline agents and refine design goals.
- Method: Participants used a "technology probe" (a simplified version of Pista) on a food warehouse restocking task.
- Outcome: Identified five key challenges organized around three "gulfs":
  - Gulf of Evaluation: Users couldn't probe agent rationale or see underlying computation logic.
  - Gulf of Execution: Users lacked confidence in editing due to unpredictable propagation of changes.
  - Gulf of Envisioning: Users struggled to specify constraints upfront and didn't know what to question.
Summative Study ( $N=16$ ):
- Goal: Compare Pista against a baseline agent (equivalent capability but no step-wise transparency).
- Method: Within-subjects design. Participants completed two financial analytics tasks (Task A: price-point lookup; Task B: weighted revenue calculation).
- Metrics:
  - Quantitative: Task success rate, number of issues detected, prompt count/length, feature usage.
  - Qualitative: Depth of verbal explanations (coding for procedural, mechanistic, and formulaic understanding), self-reported control, and trust.

3. Key Contributions

Pista System: A novel spreadsheet agent architecture that decomposes execution into auditable, steerable steps, shifting the user role from passive reviewer to active participant.
The "Semantic Diff" Primitive: The authors propose surfacing the computational unit (e.g., the formula and its logic) rather than just the surface unit (e.g., highlighted cells). This makes verification tractable by focusing on the intent behind the change.
Design Framework for Human-Agent Collaboration: The paper argues that meaningful oversight requires active participation in execution rather than improved post-hoc review mechanisms.
Empirical Evidence: Demonstrates that step-wise interaction improves user comprehension, error detection, and the sense of co-ownership without sacrificing task success rates.

4. Results

The study yielded significant findings across quantitative and qualitative dimensions:

Task Performance: Both Pista and the Baseline achieved comparable task success rates ( $M_P = 0.53$ vs. $M_B = 0.50$ ), indicating Pista does not hinder efficiency.
Error Detection: Pista users detected significantly more issues per session ( $M_P = 2.12$ ) compared to the Baseline ( $M_B = 1.75$ ).
Interaction Efficiency: Pista users required significantly fewer prompts ( $M_P = 3.18$ vs. $M_B = 4.00$ ) and shorter prompts ($12.9$ words vs. $27.6$ words). This suggests that Localized Editing allows for targeted corrections rather than full re-prompting.
Understanding Depth: Qualitative analysis of verbal explanations showed Pista users provided significantly richer explanations ( $M=4.38$ codes vs. $M=2.94$ ), covering more procedural steps, mechanisms, and formula logic.
Trust and Control:
- Users reported a significantly higher sense of control and ownership with Pista.
- Trust was described as "accumulated" through the interaction process rather than granted to the final output.
- The Branching Exploration feature was the most adopted (94% of participants), allowing users to safely experiment with alternatives.

5. Significance and Implications

Paradigm Shift: The paper challenges the prevailing "executor vs. reviewer" model in AI. It posits that for high-stakes, structured knowledge work (like finance), the user must be present during the agent's decision-making process.
Calibrated Trust: By making the reasoning process visible and editable, Pista fosters "calibrated trust." Users trust the output more because they have verified the steps, not because the output is perfect. This reduces both over-reliance (blind trust) and under-reliance (skeptical rejection).
Generalizability: While focused on spreadsheets, the "Semantic Diff" and "Participatory Execution" concepts are applicable to other domains where agents produce structured artifacts, such as code generation (surfacing function logic vs. line diffs) and document synthesis (surfacing rhetorical moves vs. character edits).
Future Directions: The authors suggest that future agentic systems should treat "envisioning support" (helping users formulate intent) as a persistent layer throughout execution, not just at the start.

In conclusion, Pista demonstrates that transparency alone is insufficient; effective human-AI collaboration in complex domains requires interactivity and presence, allowing users to steer the agent's reasoning in real-time.