💻 computer science

Click-to-Ask: An AI Live Streaming Assistant with Offline Copywriting and Online Interactive QA

This paper introduces Click-to-Ask, an AI-driven live streaming assistant that combines offline multimodal processing for generating compliant promotional copy with an online interactive module enabling streamers to efficiently address real-time viewer inquiries, thereby enhancing engagement and operational efficiency in e-commerce.

Original authors: Ruizhi Yu, Keyang Zhong, Peng Liu, Qi Wu, Haoran Zhang, Yanhao Zhang, Chen Chen, Haonan Lu

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruizhi Yu, Keyang Zhong, Peng Liu, Qi Wu, Haoran Zhang, Yanhao Zhang, Chen Chen, Haonan Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a live-streaming salesperson. You're on camera, holding a new smartphone, trying to sell it to thousands of people. It's a high-energy job, but it comes with two massive headaches:

The Prep Work: Before you go live, you have to read hundreds of pages of technical manuals, price sheets, and feature lists, then write a catchy script to sell the phone. It takes hours.
The Live Chaos: While you are selling, people are typing questions in the chat like, "Does this have fast charging?" or "Is the screen scratch-resistant?" You have to stop, find the answer in your notes, and speak it out loud. If you miss a question, you lose a sale.

"Click-to-Ask" is like hiring a super-smart, invisible AI co-pilot that sits next to you and solves both problems instantly.

Here is how it works, broken down into simple parts:

1. The "Pre-Game" Coach (Offline Module)

Think of this as your personal research assistant who works while you are sleeping.

The Problem: You dump a messy pile of files on your desk: PDFs, Word docs, photos of the product, and voice notes. It's a mess.
The AI Solution: The AI acts like a super-organized librarian. It reads all your messy files, ignores the boring fluff, and creates a neat, structured "cheat sheet" of facts about the product.
The Scriptwriter: Then, it acts like a creative writer. It takes that cheat sheet and writes a fun, exciting sales script for you.
The Safety Inspector: Finally, it acts like a strict editor. It scans the script to make sure you aren't saying anything illegal or against the rules (like promising "miracle cures" or using banned words). If it finds a problem, it fixes it before you even see the script.

Result: Instead of spending 4 hours prepping, you spend 5 minutes. You get a perfect script ready to go.

2. The "Live" Sidekick (Online Module)

Now you are live on air. This is where the magic happens in real-time.

The Problem: A viewer types, "Is the battery big?" You can't read every single comment while holding the phone and smiling.
The AI Solution: This is the "Click-to-Ask" part.
- You see a question pop up on your screen.
- You simply click on that question with your mouse or finger.
- The Magic: The AI doesn't just read the text; it looks at exactly where you clicked on the screen. It knows, "Ah, the user clicked on the comment about the battery."
- It instantly pulls the answer from your "cheat sheet" (the structured data) and your "memory" (what you just said a minute ago).
- It whispers the perfect answer to you, or even types it out for you to read.

The "Memory" Trick:
Imagine you are telling a story. If someone asks, "What did you say about the price?" you don't need to remember the whole story; you just need to remember the last few minutes.
The AI does this too. It breaks your long live stream into short "episodes" or events. If you've been talking for an hour, the AI doesn't try to remember everything from the start. It only remembers the current "scene" and the last few minutes, so it never gets confused or overwhelmed.

3. How It Learned to Be So Good

The paper mentions that the AI needed special training to understand where you clicked.

The Analogy: Imagine teaching a dog to fetch. If you just say "Fetch," the dog might run in the wrong direction. But if you point your finger at the ball, the dog knows exactly what to do.
The Tech: The researchers taught the AI by showing it thousands of pictures of live streams with "finger pointers" (visual prompts) on them. They also used a special training method (called GRPO) that rewards the AI when it gets the answer right and punishes it when it guesses wrong. This made the AI incredibly sharp at connecting your click to the right question.

The Bottom Line

Click-to-Ask is like having a super-efficient team of three working for you:

A Researcher who organizes your messy files.
A Writer who creates your sales script.
A Live Assistant who listens to your clicks and answers the audience instantly.

The Result:

For the Streamer: Less stress, less prep time, and more sales.
For the Viewer: They get instant answers, so they feel heard and are more likely to buy.
The Stats: In tests, this system was right 91% of the time at recognizing the question and gave 88% quality answers, which is a huge improvement over older methods.

It turns a chaotic, high-pressure sales job into a smooth, interactive conversation.

1. Problem Statement

Live Streaming Commerce (LSC) has transformed digital retail by blending entertainment, social interaction, and shopping. However, streamers face two critical bottlenecks:

High Preparation Costs: Crafting high-quality, compliant promotional copywriting requires significant time and effort to synthesize product information from diverse sources (PDFs, images, specs) while adhering to strict platform regulations.
Real-Time Interaction Latency: During live broadcasts, streamers struggle to rapidly access specific product details to answer viewer inquiries (often appearing as "bullet chat" or comments) in real-time. Existing AI assistants often fail to accurately link a viewer's specific question (clicked on a screen) to the correct context or product data, leading to hallucinations or irrelevant responses.

2. Methodology

The authors propose Click-to-Ask, an AI-driven assistant system divided into two tightly integrated modules: an Offline Module for content preparation and an Online Module for real-time interaction.

A. Offline Module: Automated Copywriting

This module transforms heterogeneous product inputs into structured data and generates compliant promotional scripts.

Input Processing: Accepts text, images, and files (PDF, DOC, WAV). It utilizes a proprietary MinerU engine for OCR and layout analysis of PDFs and ASR for converting audio to text.
Information Integration Module (IIM): Uses a Large Language Model (LLM) with Chain-of-Thought (CoT) prompting to extract salient product elements, eliminate redundancy, and filter noise. It fuses user-provided data with external knowledge to create a structured database entry (Name, Price, Specs, Features, Services).
Copywriting Generation Module (CGM): Powered by a Large Vision-Language Model (VLM), this module generates persuasive copy in various tones (literary, professional) based on the structured data and product images. It incorporates scenario-based narratives and emotional cues.
Prohibited-term Purification Module (PTPM): A critical safety layer that scans generated copy against a lexicon derived from platform guidelines (e.g., TikTok). It uses CoT reasoning to identify and revise/remove non-compliant terms while maintaining semantic fluency.

B. Online Module: Interactive Q&A

This module enables streamers to click on viewer questions during a broadcast to trigger instant, context-aware responses.

Click-Based Chatbot: When a streamer clicks a message, the system captures the video frame and click coordinates.
Training Data Synthesis: To teach the model to associate click coordinates with specific text regions, the authors synthesized a dataset (8K images, 32K QA pairs) by embedding question texts into images and assigning random pixel coordinates, simulating the CLEVR dataset logic.
Click-Based Learning (GRPO): The model is fine-tuned using Group Relative Policy Optimization (GRPO).
- Reward Function: Rewards are given for correctly identifying the question text at the clicked location ( $\hat{q} = q^*$ ) and generating the correct answer ( $\hat{a} = a^*$ ).
- Visual Prompts: To solve the issue of VLMs struggling to map coordinates to visual content, a mouse cursor icon is superimposed at the click point during training. This visual cue significantly improves multimodal alignment.
Event-Based Historical Memory: To handle long streams without overwhelming the model:
- Streaming Event Segmentation (SES): Asynchronously segments the video stream into logical "events" based on semantic coherence (using ViT feature similarity and optical flow).
- Knowledge Extraction Accelerator (KEA): Accelerates caption generation by selecting optimal historical caption tokens as prefixes using next-token prediction logits, reducing the computational burden of retrieving full history.

3. Key Contributions

Dual-Module Architecture: A novel system separating offline content generation from online real-time interaction, specifically designed for the LSC workflow.
Visual-Coordinate Alignment: The introduction of visual prompts (mouse cursor icons) combined with GRPO training to solve the specific challenge of grounding click coordinates to text regions in live stream frames.
Efficient Memory Management: The SES and KEA modules provide a plug-and-play solution for managing long-form video context, enabling asynchronous event segmentation and accelerated historical caption retrieval.
Compliance Automation: An integrated pipeline that not only generates copy but automatically purifies it of prohibited terms, ensuring regulatory compliance before broadcast.

4. Experimental Results

The system was evaluated on a custom dataset of 396 TikTok-style live stream frames across five categories (electronics, appliances, clothing, food, others).

Offline Performance: The PTPM reduced the average number of prohibited terms in generated copy from 4.36 to 0.36, demonstrating high efficacy in compliance filtering.
Online Performance (Question Recognition Accuracy - QRA & Response Quality - RQ):
- Baseline (Qwen2.5-VL 7B): QRA = 0.512, RQ = 0.536. Struggled with spatial mapping.
- + GRPO (Text only): Moderate improvement (QRA 0.599).
- + Visual Prompts (No GRPO): Improved to QRA 0.591.
- Full System (Visual Prompts + GRPO): Achieved state-of-the-art results with QRA = 0.913 and RQ = 0.876.
- Conclusion: The combination of visual cues and policy optimization is essential for accurate question localization and response generation.

5. Significance

Practical Application: Click-to-Ask directly addresses the operational pain points of live streamers, significantly reducing preparation time and enhancing audience engagement through instant, accurate Q&A.
Technical Advancement: The paper demonstrates that standard VLMs can be effectively adapted for specific UI-interaction tasks (click-to-ask) through targeted data synthesis and visual prompting, a finding relevant to human-computer interaction (HCI) and multimodal agents.
Scalability: The asynchronous design of the memory modules (SES/KEA) ensures the system remains responsive even during long-duration broadcasts, making it viable for real-world deployment in high-traffic e-commerce environments.

The system has been demonstrated on mobile devices and is shown to be a robust solution for the next generation of AI-assisted live commerce.