ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm
This paper introduces the COM-as-Action paradigm and the ComActor agent, which leverage the Component Object Model to transform professional software manipulation from fragile visual control into deterministic program synthesis, achieving state-of-the-art performance on the new ComCADBench for industrial CAD tasks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot to use complex professional software, like a high-end 3D design program (CAD). Currently, there are two main ways people try to do this, and both have big problems:
The "Pixel Peeping" Robot (GUI-based): This robot tries to use the software exactly like a human does. It looks at the screen, finds a button with its eyes, and clicks it.
- The Problem: Professional software screens are cluttered and messy. If the robot misses a button by a single pixel, or if the screen looks slightly different, it gets confused. If it makes a mistake early on, that error piles up, and by the time it's done, the whole project is ruined. It's like trying to build a house by blindly tapping bricks together without a blueprint.
The "Toolbox" Robot (API-based): This robot tries to use a specific remote control or command line for each specific software.
- The Problem: Every software company (like Adobe, Microsoft, or Autodesk) uses a different language for their remote controls. Some don't even let you use them at all. It's like having a different key for every single door in a city, and for many doors, you don't even have a key.
The New Idea: "The Master Key" (COM-as-Action)
The authors of this paper, ComAct, discovered a "Master Key" that almost all professional Windows software already has built-in. It's called COM (Component Object Model).
Think of COM not as a visual button on a screen, but as a direct, secret telephone line to the software's brain. Instead of telling the robot to "look for the red button and click it," you can simply say, "Draw a circle here," and the software does it instantly and perfectly.
The Big Shift:
Instead of the robot acting like a clumsy human clicking around a screen, ComAct turns the robot into a programmer. It writes a short, precise script (a set of instructions) that the software executes directly. This removes the "clumsy human" errors and allows the robot to handle complex, multi-step tasks without losing its place.
The "Gym" and the "Coach"
To prove this works, the team built three major things:
- ComCADBench (The Exam): They created the first-ever test specifically for robots using real industrial design software. It's like a driving test, but instead of a car, the robot has to build complex 3D models and engineering drawings.
- ComForge (The Gym): Training a robot to do this is hard because you need thousands of real computers running at the same time to practice. They built a massive "gym" of thousands of virtual Windows computers where the robot can practice 24/7 without crashing real machines.
- ComActor (The Student): This is their new robot agent. They didn't just teach it once; they used a three-stage training method:
- Stage 1 (Learning the Alphabet): Teaching the robot the basic syntax of the "Master Key" (COM) so it can write valid scripts.
- Stage 2 (Learning from Mistakes): Letting the robot try, fail, look at the error message, and fix its own code. It learns to say, "Oh, I tried to draw a line where there wasn't one, let me fix that."
- Stage 3 (The Art Critic): Sometimes the code works, but the 3D shape is slightly wrong (like a crooked chair). They used a special reward system that measures the exact geometry of the result. If the shape isn't perfect, the robot gets a "bad grade" and has to try again until the shape is mathematically accurate.
The Results
When they put ComActor to the test:
- Old Robots (GUI-based): They failed almost everything (0% success). They got lost in the visual clutter.
- Smart Robots (Top AI models) using the old way: Even the smartest AI models failed when trying to click buttons around.
- ComActor: It achieved state-of-the-art results. It didn't just succeed; it handled long, complex tasks (like building a part, assembling it, and then drawing a blueprint) where other robots would have collapsed. It even generalized well to other types of design tests it hadn't seen before.
The Bottom Line
The paper argues that to automate professional software, we shouldn't try to make robots look more like humans (clicking buttons). Instead, we should make them act like programmers using the software's built-in "brain interface" (COM). By doing this, and training them in a massive virtual gym with a strict "geometry coach," they can achieve a level of precision and reliability that was previously impossible.
Note: The authors explicitly state their current work is limited to CAD software on Windows. While they believe this could work for Office or Adobe software in the future, their paper only proves it for 3D design tools right now.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.