Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents
This paper evaluates a Plan Mode prototype for spreadsheet agents through a user study, finding that while it does not improve task outcomes, it effectively reduces the need for refinement and enhances users' perceptions of creativity support and human-machine collaboration.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to build a massive, complex Lego castle. You have a super-smart robot assistant who can snap bricks together faster than you can blink. But here's the catch: the robot is so eager to help that it might start building a tower exactly where you wanted a moat, or it might use red bricks when you secretly wanted blue ones. In the world of "agentic" AI—tools that act on their own to solve problems—this is a common headache. To fix this, some tools have introduced a "Plan Mode." Think of this like a blueprint session before the construction begins. Instead of the robot just grabbing a hammer and swinging, it pauses, shows you a drawing of what it thinks you want, and asks, "Did you mean this?" This gives you a chance to say, "No, actually, I wanted a drawbridge here," before a single brick is laid. But does this extra step actually help regular people, who might just want to get things done quickly, or does it just slow them down? That's the big question researchers at Microsoft set out to answer.
The paper, titled "Plans Work in Mysterious Ways," dives into whether this "Plan Mode" works for spreadsheet users. Spreadsheets are a bit different from professional coding; people who use them often like to tinker as they go, changing their minds and fixing things on the fly, rather than writing a perfect manual first. The researchers wondered: If we force a spreadsheet user to stop and make a detailed plan with an AI, will they end up with a better spreadsheet, or will they just feel like they're wasting time?
To find out, the team built a special version of an AI spreadsheet assistant that could do two things. First, it had an "Act Mode," where it just listened to your request and started building the spreadsheet immediately, letting you tweak it later. Second, it had a "Plan Mode," where the AI would stop, ask you clarifying questions (like "How many months should this budget cover?"), draw up a step-by-step plan, and let you edit that plan before it touched the spreadsheet. They then invited 24 people to try both modes on tasks like creating a personal budget or planning a vacation.
The results were a bit surprising, like finding out that a fancy GPS doesn't get you to the destination faster, but makes the ride feel much more comfortable. The researchers found that the final spreadsheets created in both modes were almost identical. Whether the users planned first or just jumped in, the end result looked very similar in terms of charts, tables, and data. The "Plan Mode" didn't magically produce more creative or complex spreadsheets.
However, the experience was totally different. People who used Plan Mode felt much more in control and felt like they were collaborating with a partner rather than just commanding a machine. The AI's habit of asking clarifying questions helped users think deeper about what they actually wanted before the work started. Even though the users rarely actually clicked the buttons to edit the plan or stop the process halfway through, just seeing the plan and having the option to change it made them feel safer and more satisfied. It was like having a co-pilot who checks the map with you; even if you don't change the route, knowing they checked it makes you feel better about the journey.
Interestingly, the study suggested that this "planning" approach works best for people who like to think things through before acting, or for longer, more complex tasks. For people who prefer to just start building and fix things as they go, the planning step felt a bit like a speed bump. But overall, the study suggests that while Plan Mode might not change the final product, it changes the relationship between the human and the AI, making the collaboration feel more like a team effort and less like a command-and-control situation. The researchers concluded that this sense of control and clarity is valuable in its own right, even if the final spreadsheet looks the same.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.