Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale

The paper introduces Sandpiper, a mixed-initiative system that integrates interactive researcher dashboards with agentic LLMs to enable scalable, privacy-preserving, and rigorous qualitative analysis of large-scale educational discourse while mitigating hallucinations and ensuring methodological consistency.

The Gist

Imagine you are a detective trying to solve a massive mystery, but instead of a few clues, you have millions of pages of conversation logs from students and teachers talking in online classrooms. You need to read every single word to understand how people learn, but your brain can only handle so much before you get tired, and reading it all by hand would take a lifetime.

This is the problem researchers face today. They have too much data and not enough time.

Enter Sandpiper. Think of Sandpiper not as a robot that replaces the detective, but as a super-smart, tireless assistant who helps the detective do their job faster and more accurately.

Here is how Sandpiper works, broken down into simple concepts:

1. The "Privacy Shield" (DG1)

Before the assistant even looks at the messy pile of papers, it puts on a pair of magic goggles. These goggles instantly blur out names, faces, and any personal secrets (like "I'm John from 3rd grade") so that the data is anonymous.

• The Analogy: Imagine a librarian who takes every book you bring in, covers the author's name and your address with a sticker, and then lets you read it. This ensures that even if the assistant makes a mistake, no one's privacy is ever compromised. The whole operation happens inside a secure, university-owned "fortress" so no outside hackers can get in.

2. The "Strict Rulebook" (DG2)

Usually, when you ask a smart AI to summarize things, it might make things up (called "hallucinations") or get confused. Sandpiper solves this by giving the AI a strict rulebook (a codebook) and a quality control inspector.

• The Analogy: Imagine you are hiring a chef to make a specific cake. Instead of just saying, "Make a cake," you give them a recipe with exact measurements. If the chef tries to put salt in the cake or forgets the eggs, an inspector (the Orchestrator) stops them immediately, says, "No, that's wrong," and sends them back to try again until the cake is perfect. Sandpiper forces the AI to follow the researcher's specific rules so it never invents fake facts.

3. The "Team Huddle" (Mixed-Initiative)

Sandpiper doesn't just let the AI do everything and walk away. It keeps the human researcher in the loop.

• The Analogy: Think of it like a co-pilot system in a plane. The AI (the co-pilot) does the heavy lifting, scanning thousands of pages and highlighting the important parts. But the Human (the pilot) is still in the seat, looking at the screen, checking the AI's work, and saying, "Yes, that's right," or "No, look at this part again." The human and the AI work together as a team.

4. The "Report Card" (DG3)

How do we know the AI is actually doing a good job? Sandpiper has a built-in scorecard.

• The Analogy: Every time the AI labels a piece of conversation, Sandpiper compares it to what a human expert would have said. It calculates a grade (like a test score) to see how often they agree. If the AI starts getting sloppy, the researcher sees the grade drop and can tweak the instructions (the "prompt") to get better results. It turns AI coding into a science experiment where you constantly improve the tool.

Why Does This Matter?

Before Sandpiper, researchers had to choose between quality (reading a few conversations carefully) or quantity (using a computer to read everything but getting it wrong).

Sandpiper gives them both. It allows researchers to analyze massive amounts of educational data with the speed of a computer but the careful, nuanced understanding of a human expert. It turns the impossible task of reading a library's worth of conversations into a manageable, trustworthy, and efficient process.

In short: Sandpiper is the bridge that lets us use powerful AI to understand human learning without losing our privacy, our rules, or our human touch.

Technical Summary

Based on the paper "Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale," here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

The rapid expansion of digital educational environments has generated an unprecedented volume of semi-structured discourse data (e.g., tutoring transcripts, classroom recordings, peer discussions). While this data holds immense potential for understanding pedagogical dynamics, analyzing it at scale presents a critical bottleneck:

• Labor Intensity: Traditional qualitative analysis relies on human experts performing close reading and line-by-line coding, which is prohibitively expensive, time-consuming, and prone to cognitive fatigue.
• Scalability Limits: Existing tools (like NVivo) struggle to handle thousands of transcripts without massive manual effort, leaving vast amounts of educational data under-analyzed.
• AI Limitations: Fully automated Natural Language Processing (NLP) and Large Language Model (LLM) approaches often lack the nuance required for complex educational discourse. They frequently suffer from "hallucinations," malformed outputs, and a failure to adhere strictly to researcher-defined qualitative codebooks, necessitating extensive data cleaning rather than analysis.
• Privacy Concerns: Using external AI models for sensitive educational data raises significant privacy and data sovereignty issues.

2. Methodology: The Sandpiper System

Sandpiper is a mixed-initiative system designed to bridge high-volume conversational data with human qualitative expertise. It treats AI as a collaborative partner rather than a replacement, utilizing a multi-tiered architecture to ensure scalability, rigor, and privacy.

Core Architecture & Workflow

The system operates through a secure pipeline connecting a researcher dashboard with an agentic LLM engine:

1. Data Ingestion & De-identification (DG1):
   • Raw transcripts are normalized into a consistent JSON "Session" format.
   • An automated, context-aware de-identification system strips or masks Personally Identifiable Information (PII) before data reaches any LLM.
   • Researchers verify masked transcripts via a human-in-the-loop interface to ensure ethical compliance.
2. Schema-Constrained Orchestration (DG2):
   • Prompt Management: Researchers define coding schemas explicitly in JSON format alongside natural language instructions within a Prompt Editor.
   • Orchestrator Loop: Background workers execute LLM requests but do not accept raw outputs blindly. An "Orchestrator" validates every output against the strict JSON schema.
   • Self-Correction: If an output is malformed or hallucinates, the orchestrator generates a feedback message detailing the error and re-prompts the LLM until the output is valid.
   • Secure Routing: All inference is routed through Cornell's LiteLM, a university-housed secure AI gateway, allowing access to closed models (e.g., GPT, Gemini) without compromising institutional data sovereignty.
3. Mixed-Initiative Interface:
   • Session Explorer: Displays original transcripts side-by-side with AI-generated codes.
   • Chat View: Allows researchers to monitor the AI's reasoning and identify edge cases for codebook refinement.
   • Run Management: Researchers define "Runs" (corpus + prompt version + model) and manage asynchronous processing.
4. Integrated Evaluation Engine (DG3):
   • Researchers can group "Runs" into "Run-Sets" to compare different models, prompt versions, or human baselines.
   • The dashboard automatically calculates qualitative metrics: pairwise agreement matrices, Cohen's Kappa, precision, and recall.

3. Key Contributions

The paper proposes three primary design goals and their corresponding technical solutions:

• DG1: Scalable De-identification and Verification:
  • Solution: Implementation of hidden-in-plain-sight, automated de-identification workflows and a secure, university-housed LLM cluster to guarantee data privacy.
• DG2: Schema-Constrained Reliability:
  • Solution: An agentic orchestration loop that enforces strict adherence to researcher-defined codebooks. This eliminates LLM hallucinations and malformed tags by scoring and rejecting non-compliant outputs before they reach the researcher.
• DG3: Verification and Benchmarking:
  • Solution: A dedicated evaluation engine that treats "prompt engineering" as a scientific activity. It enables continuous benchmarking of AI performance against human labels, fostering iterative model refinement.

4. Results

Note: As the paper is a system description and proposal (dated 2026), it does not report final empirical results from a completed user study but rather outlines the system's capabilities and the proposed evaluation framework.

• System Validation: The architecture successfully demonstrates a pipeline that isolates the researcher from heavy computational loads while maintaining data sovereignty.
• Proposed Evaluation: The authors propose a user study to evaluate the system's efficacy in:
  • Improving research efficiency.
  • Enhancing inter-rater reliability.
  • Building researcher trust in AI-assisted workflows.
• Technical Outcome: The system successfully bridges the gap between the scalability of LLMs and the methodological rigor required for educational research by preventing "black box" usage.

5. Significance

Sandpiper represents a paradigm shift in computational education research:

• Methodological Rigor at Scale: It solves the "qualitative bottleneck" by enabling the analysis of thousands of transcripts without sacrificing the nuance of human coding.
• Trustworthy AI: By enforcing schema constraints and local data routing, it addresses the two biggest barriers to AI adoption in education: hallucination risks and privacy concerns.
• Human-AI Partnership: It moves beyond the binary of "human vs. machine" to a "mixed-initiative" model where the AI handles volume and the human handles validation and refinement.
• Scientific Standardization: It elevates prompt engineering to a rigorous scientific practice, allowing researchers to systematically validate AI performance against human baselines, thereby establishing a new standard for computational qualitative analysis.

In conclusion, Sandpiper provides a scalable, secure, and rigorous infrastructure that empowers researchers to unlock the potential of massive educational discourse datasets while maintaining ethical standards and methodological integrity.