Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety

This paper introduces WebRRSBench, a comprehensive benchmark constructed from 729 websites to evaluate the reasoning, robustness, and safety capabilities of Multimodal Large Language Models (MLLMs) in web understanding, revealing significant gaps in their ability to handle compositional reasoning, UI perturbations, and safety-critical interactions.

The Gist

Imagine you are hiring a super-smart robot assistant to help you navigate the internet. You want this robot to not just "see" a website, but to truly understand it: knowing where the "Buy Now" button is, filling out your address form correctly, and, most importantly, knowing not to click the "Delete My Account Forever" button by mistake.

This paper introduces a new, rigorous driving test for these AI robots, called WebRRSBench.

Here is the breakdown of why this test was needed, what it involves, and what the results tell us, using some everyday analogies.

1. The Problem: The "Good Student" vs. The "Real World"

Previously, we tested these AI robots on simple tasks, like "What color is this button?" or "Write the code for this page." They passed these tests with flying colors.

But the real internet is messy.

• The Reasoning Gap: Imagine a robot that can read a menu but can't figure out that the "Exit" sign is above the "Menu" sign. It sees the words but doesn't understand the layout.
• The Fragility Gap: If you change the font size slightly or move a button two inches to the left, the robot panics and thinks the whole page has changed. It's like a driver who crashes because the road sign was painted a slightly different shade of blue.
• The Safety Gap: The robot might click "Delete Account" thinking it's just "Clearing Cache." It lacks the common sense to know which buttons are dangerous.

The authors realized that existing tests were like giving a driver a test drive on a perfectly empty, straight track. They needed a test that included potholes, confusing intersections, and red lights.

2. The Solution: WebRRSBench (The "Obstacle Course")

The researchers built a massive obstacle course using 729 real websites and 3,799 questions. They call it WebRRSBench, which stands for Reasoning, Robustness, and Safety.

Think of it as a three-part exam:

Part A: The Logic Puzzle (Reasoning)

• The Task: The robot is shown a webpage and asked, "Is the 'Login' button to the left or right of the 'Search' bar?" or "Fill out this form based on the user's goal."
• The Analogy: It's like asking a child to navigate a room: "Is the lamp to the left of the sofa?" Most robots, surprisingly, get this wrong. They see the objects but can't build a mental map of where they sit relative to each other.

Part B: The "Chaos" Test (Robustness)

• The Task: The researchers take a website and mess with it in three specific ways:
  1. Color Shift: They make the buttons look like they are in a fog (low contrast) or change the color of 30% of the buttons.
  2. Text Glitch: They swap an "o" for a "0" or add random exclamation marks to button text.
  3. Layout Shuffle: They move the DOM elements around (like rearranging furniture) without changing the content.
• The Analogy: Imagine you are driving, and suddenly the road signs are painted in a different color, the letters are slightly misspelled, or the traffic lights are moved to the side of the road. A robust driver (AI) should still know to stop at the red light. A fragile one crashes.

Part C: The Danger Zone (Safety)

• The Task: The robot must identify buttons that could cause permanent harm, like "Permanently Delete Account" or "Confirm Irreversible Payment."
• The Analogy: It's like a child playing with a toolbox. A safe robot knows not to touch the "Saw" or the "Drill" without permission. A dangerous one might grab them thinking they are just toys.

3. The Results: Who Passed the Test?

The researchers tested 11 different AI models (both open-source and big commercial ones like GPT-5 and Claude).

• The Big Kids Win: The expensive, closed-source models (like GPT-5) generally did the best. They were better at spotting the "dangerous" buttons and handling the messy layouts.
• The Open-Source Struggle: The free, open-source models were good at some things but struggled with the "Logic Puzzle" (spatial reasoning). They often couldn't tell which button was where.
• The "Brittleness" Issue: When the researchers messed with the colors or text, almost all models got confused. They relied too much on the look of the page (e.g., "The big red button is the important one") rather than the meaning (e.g., "The button says 'Submit'"). When the red was removed, they forgot what the button did.

4. The Good News: Training Helps!

The researchers tried a "tutoring session" (called Fine-tuning) on one of the weaker models. They gave it extra practice specifically on these tricky tasks.

• The Result: The model's performance jumped significantly. It went from getting 16% of the spatial questions right to 41%.
• The Takeaway: These models aren't hopeless; they just need specific training on how to navigate the messy, real-world web, not just the clean, textbook version.

Summary

This paper is a wake-up call. We have built AI that can "see" the web, but they are currently like tourists with a map who get lost if the street signs change color.

WebRRSBench is the new standard to ensure that before we let AI agents control our bank accounts or delete our files, they can:

1. Reason about where things are.
2. Stay calm when the website looks weird or broken.
3. Know which buttons are dangerous.

It's a crucial step toward making our digital assistants not just smart, but safe and reliable.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly deployed as core engines for web-facing systems, such as GUI agents and front-end automation tools. However, existing benchmarks (e.g., VisualWebBench, WebUIBench) primarily focus on visual perception, UI code generation, or basic element grounding. They exhibit three critical limitations when applied to real-world web automation:

1. Inadequate Reasoning Evaluation: They fail to assess spatial reasoning (positional relationships between elements) and semantic understanding of UI hierarchies, which are crucial for agents to navigate complex layouts.
2. Lack of Robustness and Safety Assessment: Existing datasets lack adversarial perturbations (e.g., layout shifts, color changes, text edits) and safety-critical scenarios (e.g., irreversible actions like account deletion). Consequently, model resilience against distribution shifts and security risks remains unexplored.
3. Limited Extensibility: Most benchmarks are static, making it difficult to programmatically expand test cases or adapt to evolving model capabilities.

2. Methodology: WebRRSBench

The authors introduce WebRRSBench, a comprehensive benchmark designed to jointly evaluate Reasoning, Robustness, and Safety across 729 real-world websites, containing 3,799 QA pairs.

A. Dataset Construction

• Source: Data is curated from existing datasets (Mind2Web, WebMMU, WebSRC) and design communities (V0 Community, top 500 sites).
• Strategy: Unlike previous works that force all tasks onto a single page, WebRRSBench filters specific pages to target specific evaluation dimensions (Reasoning, Robustness, Safety).
• Quality Control: Ground truths are established via consensus among four PhD students, ensuring high-quality annotations for spatial relationships, UI grouping, and safety risks.

B. Task Definitions (8 Tasks)

The benchmark is divided into three core dimensions:

1. Reasoning Tasks (4 Tasks):

• Position Relationship Reasoning: Determining relative spatial positions (e.g., top-left, overlap) between element pairs.
• UI Grouping: Identifying which functional region (e.g., sidebar, main content, header) an element belongs to.
• Form Filling: Inferring user goals to fill in form fields based on context.
• Hint Text Prediction: Generating appropriate placeholder/hint text for form elements based on context.

2. Robustness Evaluation (3 Perturbation Types):
The authors employ a paired evaluation protocol where models are tested on original and perturbed screenshots. Robustness is measured by the consistency of outputs.

• Color Robustness:
  • Global Low-Contrast: Reduces foreground-background contrast (simulating vision impairment).
  • Partial Button Chroma Shift: Randomly changes colors of 10–30% of buttons.
  • All-Button Chroma Shift: Changes colors of all actionable buttons.
• Text Robustness: Injects whitespace, symbols, or visually similar character substitutions (e.g., 'o' → '0') into button labels while preserving functional intent.
• Layout Robustness: Performs minimal DOM structural modifications (deleting/inserting nodes) to simulate front-end updates without altering core semantics.

3. Safety Evaluation (1 Task):

• Safety Critical Detection: Identifies UI controls that trigger irreversible or high-risk actions (e.g., "Delete Account," "Non-refundable Purchase"). The benchmark strictly excludes reversible actions (e.g., "Logout").

C. Evaluation Pipeline

• Models Evaluated: 11 MLLMs (3 closed-source: GPT-5, Claude-4-Sonnet, Gemini 2.5-Pro; 8 open-source: Qwen2.5-VL series, Pixtral, Llama4, Intern-S1).
• Metrics: Accuracy for classification tasks; embedding-based similarity for text generation; a robustness score $R = \max(0, 100 - 3\Delta)$, where $\Delta$ is the performance gap between original and perturbed inputs.
• Fine-tuning Study: The authors applied LoRA-based fine-tuning to Qwen2.5-VL-7B on specific tasks to test if targeted supervision can close performance gaps.

3. Key Contributions

1. First Unified Framework: The first benchmark to systematically evaluate MLLMs across Reasoning, Robustness, and Safety dimensions specifically for web GUI agents.
2. Novel Reasoning Tasks: Introduces spatial relationship inference and UI grouping, addressing gaps in current spatial and semantic understanding evaluations.
3. Adversarial Robustness Standards: Proposes three novel perturbation methods (color, text, layout) based on WCAG guidelines and real-world UI variations to test model resilience.
4. Safety Critical Detection: Fills a crucial gap by evaluating models' ability to recognize irreversible, high-stakes actions.
5. Extensibility: The framework supports automatic generation of new samples (especially for spatial reasoning and adversarial cases), ensuring long-term utility.

4. Experimental Results & Key Findings

Overall Performance (RQ1)

• Closed-source vs. Open-source: Closed-source models (GPT-5, Gemini 2.5-Pro) consistently outperform open-source models, particularly in Safety Critical Detection and complex reasoning.
• Reasoning Bottleneck: Position relationship reasoning and form filling remain the most challenging tasks for all models, with accuracy often below 50% even for top-tier models.
• Robustness Variability: While some large open-source models (e.g., Qwen2.5-VL-72B) show competitive robustness in color/text perturbations, they struggle significantly with layout changes and spatial reasoning.

Impact of Fine-tuning (RQ2)

Targeted LoRA fine-tuning significantly improved performance on specific tasks:

• Position Reasoning: Accuracy increased from 16.3% to 41.3% (2.5x improvement).
• UI Grouping: Accuracy rose from 67.6% to 96.9%.
• Color Robustness: Average accuracy improved from 73.1% to 80.1%.
• Conclusion: Targeted supervision effectively enhances specific web understanding capabilities.

Failure Analysis (RQ3)

The paper identifies three systematic vulnerabilities causing model failures:

1. Over-reliance on Visual Salience: Models prioritize bright colors or high-contrast blocks over textual/structural cues, leading to errors when colors are perturbed.
2. Brittleness to Text Edits: Minor character-level OCR errors (e.g., '0' vs 'o') cause large deviations in functional interpretation.
3. Spatial Attention Bias: Models tend to over-attend to local regions while neglecting global page structure, leading to incomplete summaries during layout perturbations.

5. Significance

• Benchmarking Standard: WebRRSBench establishes new standards for evaluating MLLMs in web automation, moving beyond simple code generation to complex reasoning and safety.
• Safety Awareness: It highlights the critical need for models to recognize irreversible actions, a prerequisite for deploying autonomous agents in real-world financial or account management scenarios.
• Guidance for Future Research: The findings suggest that future model development must prioritize compositional reasoning, structural understanding over visual saliency, and adversarial training to ensure robustness against real-world UI variations.
• Open Resource: The authors provide the code, dataset, and evaluation pipeline to the community to facilitate reproducible research in web understanding.