🤖 AI

MELLON - Multimodal Enhanced LLM for Online Navigation

The paper introduces MELLON, a multimodal framework that significantly enhances web navigation agent performance on the WebShop benchmark by aligning text and images to improve reasoning and planning, achieving a 9.26% accuracy increase after just one epoch of training.

Original authors: Ruiyu Li, Haoyang Cai, Zhitong Guo, Tong Hu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruiyu Li, Haoyang Cai, Zhitong Guo, Tong Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to go shopping on the internet. You might think, "Just give it the text instructions, like 'buy a red shirt,' and it will read the website and click the buttons." But here's the catch: websites aren't just walls of text. They are colorful, visual places full of pictures, layouts, and designs. If you ask a human to find a specific pair of shoes, they don't just read the description; they look at the photo to see the color, the style, and the shape. For a long time, the smartest computer programs trying to navigate the web were like blind shoppers—they could read the text perfectly but couldn't "see" the images. This paper lives in the world of Artificial Intelligence, specifically focusing on "Web Navigation Agents." These are digital helpers designed to understand natural language commands and perform tasks on real websites, like buying items or booking tickets. The big question researchers are asking is: Can we teach these agents to use both their "eyes" (to see images) and their "brain" (to read text) at the same time, just like humans do?

The researchers behind this study, working on a project called MELLON, decided to test this idea on a simulated online store called WebShop. They wanted to see if giving an AI agent a view of the product pictures, alongside the text descriptions, would help it make better decisions. They didn't just guess; they built three different experimental tools to try and solve the puzzle. The first, and most successful, was MELLON itself. Think of MELLON as a super-smart shopper who wears special glasses that translate the visual world into a language the AI's brain can understand. The team found that by training this agent for just one round of learning (one "epoch"), it got significantly better at its job. Specifically, the accuracy of completing tasks jumped by 9.26% compared to a basic version that only looked at text. This suggests that mixing vision and text is a powerful way to help AI navigate the web, even if the system still has some growing pains.

However, the story isn't a simple "multimodal is always better" victory lap. The team also tried two other creative approaches that didn't work as well as they hoped. One idea was to treat the shopping task like a "Visual Question Answering" game (VQAgent), where the AI looks at a picture and a question, then picks the right answer. They thought this would work because shopping is similar to answering questions about images. But they discovered that in the messy, real-world context of WebShop, the text descriptions actually held more important clues than the pictures did. Because the AI was trained to rely heavily on the images, it got confused when the text was the real key to the solution. It's like trying to solve a riddle by staring at a drawing when the answer is hidden in the fine print.

The third attempt involved a "Multimodal Ranker," a tool designed to make the agent pause and carefully compare every single item on a page before making a choice, rather than just grabbing the first thing it sees. The hope was that this extra thinking time would lead to better results. Instead, the experiment showed that this extra caution actually made the agent slower and less accurate. The researchers found that the tools used to measure similarity between images and text (called BERT and CLIP scores) didn't match up well with what actually made a product a "good match" in the game's scoring system. It turned out that trying to rank every single option was like a shopper spending an hour comparing every shirt in the store, only to end up buying the wrong one because they got overwhelmed.

So, what is the final takeaway from this digital shopping adventure? The paper suggests that while giving AI agents the ability to "see" the web is a promising path forward, it's not a magic wand. The MELLON agent proved that aligning images and text can boost performance, but the other experiments showed that simply adding more complex visual reasoning or forcing the agent to look at everything isn't always the right strategy. The researchers conclude that we need to keep exploring how to best mix these visual and textual clues, perhaps by training the agents longer or using even smarter "brains" (Large Language Models) in the future. For now, they've shown that a little bit of vision goes a long way, but the AI still needs to learn exactly when to look at the picture and when to read the fine print.

Technical Summary: MELLON - Multimodal Enhanced LLM for Online Navigation

Problem Statement

Web navigation agents are designed to autonomously execute tasks on websites based on natural language instructions. While current baselines often leverage Large Language Models (LLMs) for their reasoning capabilities, they predominantly operate in a unimodal fashion, relying solely on text (HTML or accessibility trees) and ignoring visual information. The authors argue that this is a critical limitation, particularly in e-commerce environments like WebShop, where visual cues (e.g., product finish, color, style) often contain qualitative information absent from textual descriptions. Existing multimodal approaches either lack strong reasoning abilities or struggle with the dynamic, action-constrained nature of web navigation, where deviations from a predefined action space lead to task failure.

Methodology

The paper proposes three distinct multimodal approaches to address these challenges, with a primary focus on MELLON (Multimodal Enhanced LLM for Online Navigation).

1. MELLON (Multimodal Enhanced LLM for Online Navigation)

MELLON is a web agent designed to integrate visual and textual modalities for enhanced reasoning and planning. Its architecture consists of three core innovations:

Better Encoders: The model utilizes a pre-trained Vision Transformer (ViT) backbone coupled with a Q-Former to generate image encodings. Unlike traditional CNNs (e.g., ResNet), the ViT treats image patches as tokens, capturing spatial relationships more effectively.
Text-Image Alignment and Efficient Training: To bridge the gap between the visual encoder and the LLM, a single projection layer is employed. This layer aligns visual embeddings with the LLM's input space.
- Training Strategy: The authors employ a parameter-efficient approach where the LLM (CodeLlama) and the ViT are frozen. Only the projection layer and the Q-Former are trained.
- Loss Function: The model is trained using a cross-entropy generation loss on pairs of multimodal inputs and successful actions. The WebShop completion score is integrated into the loss function to jointly optimize reasoning and action alignment.
Extended ReAct Prompts: The standard ReAct (Reasoning + Acting) prompt is redesigned to include aligned visual representations (inserted via <ImageHere>). The prompt engineering was heavily optimized to fit within 24 GB GPU memory constraints, ensuring the agent can perform sparse reasoning and select actions from a valid space ( $A' = A \cap L$ ).

2. VQAgent

This approach reframes the WebShop task as a multi-hop Visual Question Answering (VQA) problem, where "answers" correspond to actions.

Architecture: It utilizes the BEiT3 model, a state-of-the-art vision-language model.
Adaptation: Unlike standard VQA (a generation task), WebShop is treated as a classification task. The model encodes the observation (image, product description, instruction) and all possible actions separately using BEiT3. These embeddings are then processed through a bi-attention layer and a linear layer to score actions.
Training: The model is trained via Imitation Learning on human trajectories to mimic the correct action selection.

3. Multimodal Ranker

This component aims to improve exploratory behavior by systematically evaluating all available options before making a decision, rather than selecting the top initial result.

Mechanism: It combines BERT scores (measuring textual similarity between instructions and product titles) and CLIP scores (measuring image-text congruence).
Training: A learnable parameter balances these two dimensions, and the ranker is trained using Mean Squared Error (MSE) loss against the WebShop completion score as a label.

Key Results

MELLON Performance

Accuracy Improvement: Training MELLON for just one epoch resulted in a 9.26% increase in step-wise task completion accuracy compared to the unimodal ViT+CodeLlama inference baseline (rising from 5.11% to 14.37%).
Ablation Studies:
- Replacing the ResNet baseline with a ViT + Trainable QFormer yielded comparable scores (39.92) but slightly lower success rates (9.6% vs 9.8%) compared to ResNet, suggesting that simpler models may suffice in certain contexts or that the task requires limited multimodal context.
- The use of CodeLlama was preferred over Vicuna-7B due to fewer hallucinations and better adherence to task formats, though CodeLlama still occasionally produced empty or invalid outputs.
Qualitative Analysis: In specific cases (e.g., selecting a "wood finish" loveseat), MELLON successfully utilized visual cues to identify the correct product, whereas the unimodal baseline failed due to a lack of visual comprehension.

VQAgent and Multimodal Ranker Performance

VQAgent: This model underperformed compared to the baseline, achieving a score of 31.26 and a success rate of 5.7% (vs. baseline 39.92 and 9.8%). The authors attribute this to a mismatch in information density: in WebShop, textual descriptions often contain more critical information than images, unlike standard VQA tasks where visual information is dominant.
Multimodal Ranker: The integration of the ranking algorithm decreased performance (Score: 28.13; Success Rate: 8.6%). The authors suggest that exhaustive exploration is inefficient in WebShop's vast search space and that BERT/CLIP similarity scores do not reliably correlate with the specific reward criteria of the WebShop environment.

Significance and Claims

The paper claims to be the first to:

Develop a multimodal web navigation agent that incorporates LLMs despite the engineering challenges of mapping open-ended generation to dynamic web action spaces.
Align visual and textual representations via a projection layer into an LLM, demonstrating an efficient training strategy (freezing the LLM and ViT).
Enable strong reasoning and planning in a multimodal setting by re-designing ReAct prompts to include visual representations.

The authors emphasize that while MELLON shows significant promise with minimal training (one epoch), the results highlight the necessity for further exploration into multimodal approaches. They acknowledge current limitations, including the frozen nature of the LLM and ViT, and the engineering challenges posed by the WebShop environment. The work suggests that with more extensive training, larger memory resources, and potentially fine-tuning the LLM, multimodal agents could achieve substantially higher effectiveness in web navigation tasks.