OpeFlo: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding

OpenFlo is an automated agent that simulates human web interactions using multimodal GUI grounding to perform real-time, end-to-end usability evaluations and generate comprehensive UX reports, offering a scalable alternative to traditional user studies.

The Gist

Imagine you've just built a brand-new digital storefront. It looks great, the buttons work, and the code is perfect. But before you open the doors, you need to know: Is it actually easy for a real person to use?

Traditionally, answering this question is like hiring a film crew, renting a studio, and paying actors to try out your store while you watch them through a one-way mirror. It's expensive, slow, and you can only test a few people at a time.

OpenFlo is a new tool that changes the game. Think of it as a super-smart, tireless "Digital Twin" of a human user that you can deploy instantly to test your website.

Here is how OpenFlo works, broken down into simple concepts:

1. The "Eyes" vs. The "Code" (Visual Grounding)

Most automated testing bots are like blind people reading a Braille manual (the website's code, or DOM). They know where a button is supposed to be in the code, but they don't actually see the screen. If the code says "Button" but the screen shows a broken image or a confusing layout, the blind bot keeps clicking blindly.

OpenFlo is different. It has eyes.

• The Analogy: Imagine a bot that doesn't just read the recipe; it actually looks at the kitchen. If a sign says "Push," but the door is painted over and looks like a wall, OpenFlo sees the wall and gets confused just like a human would. It uses visual grounding to "see" the website exactly as a human does, noticing clutter, bad colors, or buttons that look disabled.

2. The "Think Aloud" Protocol (The Inner Monologue)

When you use a website and get stuck, you might mutter, "Wait, why isn't this working? Did I miss a step?"
OpenFlo does the same thing. It doesn't just click and fail; it talks to itself in real-time.

• The Analogy: It's like having a test subject wear a microphone. As it navigates your site, it narrates its thoughts: "I see the 'Checkout' button, but it looks grayed out. I'm confused. Do I need to fill out the address first?"
• This gives developers the "Why" behind the failure, not just the fact that it failed.

3. The "Report Card" (Metrics)

After the test, OpenFlo doesn't just say "It worked" or "It broke." It gives you a detailed report card using two famous grading systems:

• The "Step-by-Step" Grade (SEQ): After every single click, it asks, "How hard was that specific step?" (1 = Very Hard, 7 = Very Easy). This helps you find the exact moment a user gets frustrated.
• The "Overall Report Card" (SUS): At the end, it gives the whole website a score out of 100, similar to a school grade (A+, B, C, etc.), telling you if the site is generally usable or a disaster.

4. The "Expert Imitator" (Experience-Imitation Planning)

Sometimes, a website is tricky. Maybe the "Help" button is hidden in the footer, or you have to click three different menus to find a form.

• The Analogy: A normal bot might just click randomly. OpenFlo, however, can do a quick "Google search" on how real experts usually navigate similar sites. It learns the strategy of a pro user before it even starts clicking. It's like sending a test driver who has already studied the map of the city, rather than someone guessing the turns.

Why Does This Matter?

In the past, only big companies with huge budgets could afford to test their websites constantly. Small teams or solo developers often launched products that were technically working but terrible to use.

OpenFlo is the "Continuous Testing" revolution.

• For Developers: It's like having a personal quality control inspector who works 24/7. You can change your website, run OpenFlo, get a report, fix the issue, and run it again—all in minutes.
• For Users: It means the websites you visit will be smoother, less confusing, and more intuitive because the bugs are caught by these "Digital Twins" before real humans ever get frustrated.

In short: OpenFlo is a robot that sees, thinks, talks, and grades your website just like a human would, but it does it faster, cheaper, and without needing a coffee break.

Technical Summary

1. Problem Statement

The rapid pace of modern software development, driven by AI-assisted tools and non-professional developers, has outpaced the ability to conduct rigorous User Experience (UX) testing. Traditional UX evaluation relies on resource-intensive methods (e.g., laboratory studies, participant recruitment, manual analysis) that are too slow for agile workflows and often neglected by startups or open-source projects.

Existing automated solutions suffer from two main limitations:

• DOM-Based Limitations: Most web agents rely on Document Object Model (DOM) parsing, which ignores visual styles, layout, and accessibility issues. They "see" code rather than pixels, missing the visual clutter and ambiguity real users face.
• Lack of Human-Like Perception: Current agents often lack the ability to simulate the cognitive process of a human user (e.g., confusion, hesitation) or provide qualitative insights into why a task failed, offering only functional correctness rather than usability validity.

2. Methodology: The OpenFlo Framework

OpenFlo is an open-source agent built upon the Avenir-Web framework. It simulates human web interaction by combining Multimodal Large Language Models (MLLMs) with GUI Grounding techniques to act as a "synthetic user."

Core Architecture Components

1. Visual Perception & Grounding (MoGE):

   • Instead of relying solely on HTML/DOM, OpenFlo uses a Mixture of Grounding Experts (MoGE).
   • It overlays numerical tags on interactive elements within screenshots, allowing the agent to interact directly with pixels (coordinates).
   • This enables the agent to perceive visual layout, styles, and obfuscated elements that DOM-based agents miss.

2. Core Agent & Reasoning Loop:

   • The central MLLM (e.g., Gemini-3-Pro) operates in a closed loop: Perception $\rightarrow$ Reasoning $\rightarrow$ Action.
   • It ingests grounded screenshots and task states to generate high-level plans (e.g., "click search") translated into low-level browser actions (e.g., click(234, 550)).

3. Adaptive Memory & Checklist:

   • Maintains long-term context via an adaptive memory module and a dynamic checklist.
   • Tracks progress against subgoals, allowing the agent to recover from errors and avoid repetitive loops.

4. Experience-Imitation Planning (EIP):

   • Before execution, the agent performs web searches to retrieve procedural knowledge (e.g., help docs, forums).
   • This allows the agent to emulate the strategies of informed human experts rather than relying solely on internal training data.

The Evaluation Pipeline

OpenFlo mimics professional usability studies through three integrated phases:

• Think Aloud Protocol: The agent verbalizes its internal reasoning, interpretation of the UI, and confusion in real-time before taking action. This provides qualitative data on the "why" behind errors.
• Step-wise Single Ease Question (SEQ): After every interaction step, the agent rates difficulty, efficiency, clarity, and confidence on a 1–7 scale. This creates a granular "friction map" of the user journey.
• System Usability Scale (SUS): Upon task completion (or failure), the agent answers the standard 10-item SUS questionnaire based on the entire session memory to generate a standardized usability score.

Automated Analysis

A final MLLM acts as a UX Researcher, synthesizing the interaction logs, Think Aloud transcripts, and quantitative scores (SEQ/SUS) into a structured report. It correlates drops in SEQ scores with specific verbalizations to diagnose root causes and provide actionable design recommendations.

3. Key Contributions

• OpenFlo Agent: An open-source system capable of performing end-to-end web tasks for UX evaluation, bridging the gap between functional testing and human-centric insights.
• Advanced Evaluation Framework: A novel integration of standard metrics (SUS, SEQ) with LLM-analyzed Think Aloud reasoning, moving beyond post-hoc analysis to real-time, step-wise evaluation.
• Visual Grounding Emphasis: Demonstrates the critical importance of visual grounding (MoGE) for accurate evaluation, proving that agents must "see" the interface like humans to detect usability pitfalls.
• Empirical Validation: A case study validating the framework's effectiveness in identifying specific friction points that traditional tools miss.

4. Results & Case Studies

The authors validated OpenFlo using two distinct case studies:

Case Study A: Recreation.gov (Permit Booking)

• Task: Check permit availability for a group of 4 at Brooks Camp.
• Findings: The agent successfully navigated initial steps (SEQ 7) but encountered severe friction during date selection and group size configuration (SEQ dropped to 1).
• Insight: The DOM showed elements as "visible," but the visual interaction failed due to state desynchronization.
• Outcome: The system generated a SUS score of 55.0/100 (Grade D), correctly identifying the interface as having significant usability issues despite functional code.

Case Study B: Discogs (Documentation Search)

• Task: Locate submission guidelines, bypassing the commercial marketplace interface.
• Findings: The agent utilized EIP to ignore the search bar and navigate to the footer/help section, mimicking an expert user.
• Outcome: The agent completed the task in 4 steps with an Average SEQ of 6.0 and a SUS score of 87.5 (Grade A+).
• Significance: Demonstrated the agent's ability to handle information hierarchy and visual noise where DOM-based agents typically fail.

5. Significance and Future Work

Significance:
OpenFlo represents a paradigm shift from "can the bot click the button?" to "can the bot use the website like a human?" By grounding interactions in visual perception and synthesizing quantitative metrics with qualitative reasoning, it enables continuous, scalable, and data-driven usability testing. This empowers developers to integrate high-fidelity UX evaluation directly into the CI/CD pipeline.

Future Directions:

• Continuous Operations: Moving from discrete "pause-think-act" cycles to fluid, real-time interaction.
• Exploratory Autonomy: Developing "free-roaming" agents that can identify bottlenecks without predefined scripts.
• Domain-Specific Fine-tuning: Creating models specifically optimized for UI evaluation nuances.
• Diverse Personas: Simulating users with varying digital literacy and accessibility needs.
• Longitudinal Studies: Tracking usability changes across product iterations over time.

In conclusion, OpenFlo provides a robust, automated solution to the "usability gap" in modern software development, offering a scalable alternative to traditional user studies while maintaining the depth of human-centric evaluation.