AnchorNote: Exploring Speech-Driven Spatial Externalization for Co-Located Collaboration in Augmented Reality

This paper introduces AnchorNote, a co-located AR system that converts spoken ideas into spatially anchored digital sticky notes via live transcription and LLM summarization, revealing through user studies that while it reduces writing effort, it fundamentally reshapes collaborative cognition by introducing new coordination costs and altering how participants formulate and organize ideas.

The Gist

Imagine you and a friend are brainstorming ideas for a new project. You're sitting in a room, talking excitedly, and you both reach for a stack of yellow sticky notes. You write down a quick thought, stick it on the wall, and move on. It's messy, but it works. You can rearrange the notes, group them, and see the whole picture.

Now, imagine you are wearing smart glasses (like futuristic sunglasses) that can hear you and turn your spoken words into those sticky notes instantly, without you ever picking up a pen. That is the core idea behind a research project called AnchorNote.

Here is the story of the paper, explained simply:

🧠 The Big Idea: "Talk, Don't Write"

The researchers wanted to see if we could replace the physical act of writing with just talking. They built a system where:

1. You speak your idea.
2. The glasses listen, type it out, and use AI to give it a short title (like a headline).
3. A digital "sticky note" appears floating in the air in front of you.
4. Your friend, wearing their own glasses, sees that same note floating in the exact same spot in the room.

It sounds like magic, right? But the researchers found that while it saves you from writing, it introduces some new, tricky problems.

🎭 The Experiment: Two Rounds of Testing

They tested this with 20 students in two rounds:

Round 1: The "Wild West" (Gesture Control)

• How it worked: To create a note, you had to make a specific hand gesture in the air.
• What happened: It was chaotic.
  • The "Wait, Did I Say That?" Problem: Because the note appeared immediately as you spoke, people felt pressure to have their ideas perfectly formed before they opened their mouths. They stopped "thinking out loud" and started "performing" their thoughts.
  • The "Ghost Gesture" Problem: Sometimes, the glasses thought you made a gesture when you didn't, or they missed a real one.
  • The "Oops" Problem: If the computer misheard you (e.g., it wrote "open sesame" instead of "too much work"), you couldn't just cross it out. You had to delete the whole note and start over, which broke the flow of conversation.

Round 2: The "Controlled Zone" (Button Control)

• How it changed: They replaced the hand gestures with a simple button press on the glasses. They also added clear lights to show when the system was listening vs. when it was summarizing.
• What happened: It got much better.
  • People felt more in control.
  • They could easily delete bad notes.
  • The conversation flowed more naturally because they weren't constantly worried about accidentally triggering the system.

⚖️ The Trade-Offs: What We Gained vs. What We Lost

The paper concludes that AnchorNote isn't a perfect replacement for paper yet, but it teaches us some valuable lessons:

The Good (The Magic)	The Bad (The Glitches)
No More Pen Pains: You don't have to stop talking to write. Your hands are free to gesture and point.	The "Monitor" Burden: Instead of worrying about your handwriting, you now worry about the computer. You have to constantly check: "Is it listening? Did it hear me right?"
Shared Vision: Both people see the notes in the same 3D space, making it easy to point at an idea together.	The "Instant" Pressure: Because the note appears instantly, people feel they can't "think out loud" or say half-baked ideas. They feel they must be perfect before speaking.
AI Summaries: The system turns long rambles into short, neat titles automatically.	The Clutter: Without an easy way to delete, the room gets filled with digital junk, making it hard to see the big picture.

🌟 The Main Takeaway

The researchers found that speech-driven tools change how we think.

When you write on paper, you can scribble a messy thought, cross it out, and try again. It feels safe. When you talk to a computer that instantly turns your words into a permanent note, you feel like you are on a stage. You become more careful, less spontaneous, and more focused on the tool than the idea.

The Lesson for the Future:
If we want these smart glasses to work for teams, we can't just make them "hands-free." We need to make them forgiving. We need:

1. Draft Modes: Let us say messy things without them becoming permanent notes immediately.
2. Easy Erasers: If the computer misunderstands, we need to be able to fix it instantly without breaking the conversation.
3. Clear Signals: We need to know exactly when the computer is listening so we don't feel like we're talking to a wall.

In short: AnchorNote is a cool prototype that shows us the future of collaboration, but it reminds us that technology needs to adapt to us, not the other way around.

Technical Summary

1. Problem Statement

While physical sticky notes remain a dominant medium for collaborative brainstorming due to their low-friction nature and ability to support spatial organization, they have inherent limitations:

• Fragility of Spoken Ideas: Verbal contributions in conversation are often lost if not explicitly captured.
• Writing Burden: The need to physically write notes interrupts the flow of conversation and cognitive processing.
• Uneven Documentation: Responsibility for recording ideas often falls unevenly on one person.

Existing Augmented Reality (AR) solutions for co-located collaboration often prioritize automation or single-user productivity, lacking support for intentional, shared speech capture where turn-taking and shared attention are central. There is a gap in understanding how speech-driven, spatially persistent capture on head-worn AR devices reshapes collaborative cognition and coordination.

2. Methodology

The authors conducted a two-phase iterative design investigation involving 20 undergraduate participants working in pairs.

• System Prototype (AnchorNote):

  • Platform: Runs on Snap Spectacles AR glasses using Lens Studio.
  • Core Functionality: Users capture spoken ideas which are transcribed via Automatic Speech Recognition (ASR), summarized into a short title by a Large Language Model (LLM), and anchored as digital sticky notes in a shared physical space.
  • Synchronization: Uses the Spectacles Sync Kit (SSK) to share note instantiation, position, rotation, and ownership across devices in real-time.
  • Fallback Mechanism: Includes a local keyword-based fallback for note creation if the remote LLM service fails or times out.

• Phase 1 (Diagnostic):

  • Participants: 12 participants (6 pairs).
  • Design: Gesture-triggered note creation, no deletion mechanism, and no explicit system state indicators.
  • Task: Brainstorming and thematic grouping using analog sticky notes vs. AnchorNote.
  • Goal: Identify breakdowns, coordination costs, and attention shifts.

• Phase 2 (Reparative):

  • Participants: 8 returning participants (4 pairs from Phase 1).
  • Design Updates: Replaced gestures with an explicit button trigger, added a "hold-to-delete" mechanism, and introduced clear system-state indicators (e.g., "Recording," "Summarizing").
  • Goal: Evaluate if targeted interaction changes repaired the breakdowns identified in Phase 1.

• Data Collection: Audio/video recordings, post-task surveys (7-point Likert scale), and semi-structured interviews analyzed via thematic analysis.

3. Key Contributions

1. AnchorNote System: A novel co-located AR system that transforms spoken contributions into spatially anchored, shared artifacts using live transcription and LLM summarization.
2. Empirical Study of Speech-Driven Capture: The first study to examine how shifting from manual writing to speech-driven capture in AR alters collaborative dynamics, specifically focusing on the trade-offs between reduced writing effort and increased system monitoring.
3. Design Implications for Co-Located AR: The paper moves beyond "hands-free" novelty to argue that interaction legibility and fast error recovery are more critical for successful collaboration than automation alone.

4. Key Results & Findings

A. Reshaping Ideation Strategies

• Pre-formulation Pressure: Immediate speech-to-note conversion caused participants to feel pressure to finalize ideas before speaking. Unlike analog notes where one can "think while writing," AR participants felt they had to have a "finalized" idea to avoid recording a "draft."
• Suppression of Tentative Ideas: This shift sometimes suppressed tentative contributions and altered conversational rhythm, as users feared the permanence of the digital artifact.

B. Shift in Cognitive Load: Writing vs. Monitoring

• Reduced Writing: Participants appreciated not having to interrupt thought processes to write.
• Increased Monitoring: In Phase 1, cognitive load shifted from writing to monitoring system state. Users spent significant mental energy ensuring the system was listening, checking for transcription errors, and managing the lack of editability.
• Recovery Costs: When errors occurred (e.g., mistranscription), the inability to edit directly forced users to delete and re-record, disrupting flow.

C. Interaction Legibility and Coordination

• Gesture Ambiguity: Phase 1's gesture controls caused accidental activations and confusion about system state, leading to social awkwardness and disrupted turn-taking.
• Explicit Controls: Phase 2's button-based controls and clear status indicators significantly reduced accidental captures and made coordination easier. Participants noted that while less "hands-free," the system was easier to coordinate around, allowing it to recede into the background of collaboration.

D. Spatial Persistence and Curation

• Shared Reference: Spatially anchored notes were highly valued for shared reference and 3D organization.
• Clutter vs. Curation: Without a delete mechanism (Phase 1), the workspace quickly became cluttered, undermining sensemaking. The addition of lightweight curation (delete) in Phase 2 was essential to maintaining the utility of spatial persistence.

E. Quantitative Trends

• Mental Effort: Perceived mental effort was highest in Phase 1 (4.17) compared to Analog (2.92) and Phase 2 (3.63).
• Interruptions: Phase 1 had the highest interruption scores (5.42), primarily due to system errors and coordination needs, compared to Analog (2.67).
• Conversational Smoothness: Analog was rated highest (6.00), followed by Phase 2 (4.88), with Phase 1 being the lowest (4.08).

5. Significance and Design Implications

The study concludes that speech-driven spatial externalization is not a neutral input channel; it fundamentally restructures collaborative cognition.

• Provisionality is Key: Designers must support "draft states" or delayed publishing to allow for tentative ideation without the pressure of immediate permanence.
• Legibility over Automation: For co-located AR, clear system state indicators and explicit controls are more valuable than "magic" gesture automation. Users need to know exactly when the system is capturing to maintain conversational flow.
• Curation Mechanisms: Spatial persistence is only beneficial if users can easily curate (delete/organize) the workspace to prevent clutter.
• Future Directions: Future tools should focus on robust error recovery, multi-speaker differentiation, and accessible input modalities that do not disadvantage users with motor impairments.

In summary, AnchorNote demonstrates that while AR can reduce the physical burden of note-taking, it introduces new cognitive and social coordination costs that must be carefully managed through intentional interaction design.