🤖 AI

MetaCues: Enabling Critical Engagement with Generative AI for Information Seeking and Sensemaking

The paper introduces MetaCues, a generative AI tool designed to foster critical engagement through metacognitive cues and note-taking, which an online study of 146 participants found increased confidence in judgments and broadened inquiry, particularly for less controversial topics.

Original authors: Anjali Singh, Karan Taneja, Zhitong Guan, Soo Young Rieh

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anjali Singh, Karan Taneja, Zhitong Guan, Soo Young Rieh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery. In the past, you would have to dig through a mountain of dusty books, read every page, and piece the clues together yourself. That was hard work, but it made your brain strong.

Now, imagine you have a super-smart robot assistant (Generative AI) who can read that entire mountain of books in a second and give you a perfect summary. It's fast and convenient. But here's the catch: because the robot does all the heavy lifting, you might stop thinking for yourself. You might just accept its answer as truth without checking if it's right, or without looking for other sides of the story. This is called "cognitive offloading"—basically, letting the robot do your thinking for you.

The Problem: The "Yes-Man" Robot
The paper explains that these AI tools are so smooth and confident that they can trick us. They might agree with whatever you already believe (a bit like a "yes-man" friend who never challenges you), which stops you from seeing the whole picture. If you just accept the robot's answer, you might end up with a shallow understanding or even believe false information.

The Solution: MetaCues (The "Thinking Coach")
To fix this, the researchers built a tool called MetaCues. Think of MetaCues not as a replacement for the robot, but as a thinking coach sitting next to you while you work with the robot.

Here is how it works:

The Chat: You ask the AI your questions, just like normal.
The Coach's Nudge: While you are chatting and taking notes, MetaCues pops up with little "nudges" or reminders. These aren't answers; they are questions designed to wake up your brain.
- Example: "Hey, you haven't clicked on any of the source links yet. Maybe check one to see where that info came from?"
- Example: "You've been asking similar questions. Have you thought about what the opposite side of this argument would look like?"
- Example: "Great job taking notes! Now, try to summarize what you've learned in your own words."

The Experiment: The Music vs. Social Media Test
The researchers tested this tool with 146 people. They gave them two different topics to research:

Topic A (Social Media): "Should kids under 16 be banned from social media?" (This is a hot, controversial topic that people already have strong opinions about).
Topic B (Music): "Should students be allowed to listen to music during exams?" (This is a calmer topic that most people haven't thought much about).

What They Found
The results were like a story about two different kinds of students:

For the Music Topic (The New Territory):
When people researched the music topic, the "Thinking Coach" (MetaCues) worked like magic. Because they didn't know much about it to begin with, the nudges helped them explore wider. They asked more diverse questions, looked at more sources, and didn't just stick to one narrow idea. It was like giving a hiker a compass when they were in a new forest; they explored much more of the woods.
For the Social Media Topic (The Familiar Territory):
When people researched social media, the coach didn't change their behavior as much. Why? Because they already had strong opinions. It's like trying to convince a die-hard sports fan to watch a different team; they are already set in their ways. The nudges didn't break through their existing beliefs as easily.
The Confidence Boost:
Interestingly, people using MetaCues felt more confident in their final opinions, regardless of the topic. It seems that by forcing them to pause, reflect, and check sources, they felt like they had done a better job of "doing their homework." They felt their conclusions were more solid, even if we don't know yet if those conclusions were actually more correct.

The Big Takeaway
This paper teaches us that while AI is a powerful tool, we can't just let it drive the car. We need a "co-pilot" (MetaCues) to remind us to keep our hands on the wheel, check the map, and look out the window for other roads.

Without the coach: We might get lost in a narrow path or believe whatever the AI says.
With the coach: We are more likely to explore new paths, check our facts, and feel good about our decisions—especially when we are learning something new.

The researchers hope that in the future, all AI tools will come with this kind of "thinking coach" built-in, helping us stay smart and critical even as we rely on machines to find our information.

1. Problem Statement

The rapid adoption of Generative AI (GenAI) search tools (e.g., Perplexity.ai, Google AI Overview) has introduced significant risks to information literacy. While these tools offer speed and synthesis, their design inherently encourages cognitive offloading, where users delegate cognitive processes (understanding, analyzing, evaluating) to the AI. This leads to:

Passive engagement: Users accept AI outputs without critical verification.
Informational homogenization: Limited exposure to diverse perspectives.
Overconfidence: The fluent and confident tone of GenAI can inflate user confidence in potentially biased or hallucinated information, reinforcing confirmation bias, especially on controversial topics.

Effective use of GenAI requires metacognitive engagement (thinking about thinking) to craft prompts, verify sources, and regulate learning. However, existing solutions often rely on manual intervention (Wizard-of-Oz studies) to provide reflective cues, leaving a gap in understanding how to autonomously generate and deliver these cues within a live search environment.

2. Methodology

System Design: MetaCues

The authors developed MetaCues, a web-based interactive tool designed to act as a "learning companion." It features three main components:

Chat Interface: Powered by OpenAI's GPT-4o with web search capabilities. It is instructed to act as a teaching assistant, providing responses with citations (5+ sources), structured in Markdown, and at a Bachelor's level.
Notepad: A space for users to take notes and synthesize information.
Metacognitive Cue Engine: The core innovation. It analyzes user chat history, notes, and click-stream data to proactively deliver tailored cues.

Cue Generation Logic:
The system utilizes a hybrid approach of fixed-interval and dynamic triggers based on seven cue types derived from prior work (Singh et al.) and critical thinking indicators:

Fixed: Orienting (at session start) and Monitoring (after first query).
Dynamic (Triggered by behavior):
- Source Engagement (SE): Encourages clicking linked sources.
- Independent Thinking (IT): Encourages note-taking and reflection.
- Persistent Inquiry (PI): Encourages follow-up questions.
- Broadening Perspectives (BP): Encourages exploring unexplored viewpoints.
Adaptive Variants: Cues come in "Regular" (prompting missing behaviors) or "Reinforcement" (acknowledging existing good behaviors) variants. The system uses an LLM to determine which variant to send based on user data, though the message templates are predefined to ensure consistency.
Delivery Mechanism: Cues are queued and displayed during natural pauses (3-second idle) to minimize distraction, with a pulsing visual indicator.

Study Design

Type: Online between-subjects factorial experiment.
Participants: $N=146$ (after attention checks) recruited via Prolific; US-based, college-educated.
Conditions:
- Baseline: GenAI tool with chat and notepad, no cues.
- MetaCues: Identical tool with metacognitive cues.
Tasks: Two information-seeking tasks (25 minutes each) requiring exploration of diverse perspectives:
1. Social Media: "Should social media be banned for individuals under 16?" (High familiarity, high controversy).
2. Music: "Should students be allowed to listen to music during exams?" (Lower familiarity, lower controversy).
Metrics:
- Behavioral: Search duration, time outside interface (on sources), query count, click-through rate, and Query Divergence (semantic diversity of queries measured via cosine distance of vector embeddings).
- Outcome: Attitudinal judgments and confidence in those judgments (5-point Likert scales).

3. Key Contributions

Autonomous Metacognitive Scaffolding: Demonstrated the feasibility of a fully automated system that generates and delivers metacognitive cues in real-time based on user interaction data, moving beyond manual Wizard-of-Oz setups.
Behavioral Adaptation: Introduced a logic system that tailors cues not just to the type of behavior (e.g., lack of source clicking) but also to the state of the user (Regular vs. Reinforcement variants), addressing user preference for recognition over redundant prompting.
Topic-Dependent Efficacy: Provided empirical evidence that the effectiveness of metacognitive cues is mediated by the user's prior familiarity and the controversy level of the search topic.

4. Results

Search Behavior

Query Divergence (Exploration):
- Music Topic (Low Familiarity/Controversy): MetaCues users showed significantly higher query divergence ( $p=0.045$ ) compared to Baseline. UMAP visualizations confirmed MetaCues users explored a broader semantic space.
- Social Media Topic (High Familiarity/Controversy): No statistically significant difference in query divergence between conditions.
Source Engagement: MetaCues users showed higher (but not statistically significant) means for time spent on external sources and click-through rates across both topics.
Other Metrics: No significant differences were found in search duration, total typing time, or number of queries.

Attitudinal Judgments & Confidence

Confidence: A two-way ANOVA revealed a significant main effect of condition ( $p=0.035$ ). Participants using MetaCues reported higher confidence in their attitudinal judgments compared to the Baseline group, regardless of the topic.
Judgment Accuracy: The study did not measure the factual accuracy of the judgments, only the confidence level.

5. Significance and Discussion

Contextual Effectiveness: The study suggests that metacognitive cues are most effective at broadening inquiry when users have lower prior familiarity and the topic is less controversial. For highly familiar or polarized topics (like Social Media), users may have entrenched views that resist the influence of automated cues.
Epistemic Grounding: The increase in confidence among MetaCues users suggests that the cues successfully fostered deeper engagement and reflection, leading to a stronger sense of epistemic grounding. However, the authors caution that higher confidence does not automatically equate to higher accuracy; future work must verify if this confidence is well-calibrated.
Future Directions: The authors recommend larger sample sizes to increase statistical power, qualitative analysis of interaction patterns, and the development of more adaptive cue scheduling (e.g., fading cues as users internalize behaviors).

Conclusion: MetaCues represents a significant step toward mitigating the passive consumption of GenAI. By automating metacognitive scaffolding, it can effectively promote broader inquiry and more confident judgment formation, particularly in exploratory search tasks where users lack established mental models.