Technical Summary: MonitrLLM

Problem Statement

Current Large Language Model (LLM) evaluation infrastructures suffer from a critical disconnect between interaction data and user-defined outcomes. Existing approaches fall into three categories, each with significant limitations:

1. Benchmark Suites: Assess capabilities on controlled, predefined tasks but fail to capture open-ended, goal-driven work where users adapt strategies and negotiate constraints.
2. Large-Scale Conversation Corpora: Capture naturalistic use (e.g., OpenAssistant, WildChat) but treat conversations as generic data points, lacking user intent, task purpose, or community-specific definitions of success.
3. In-Interface Feedback: Mechanisms like "thumbs up/down" or optional text fields record momentary satisfaction but do not capture whether the output fulfilled the user's specific goal or the effort required to achieve it.

Consequently, there is no routine infrastructure that links full interaction trajectories to user-reported task intent and outcome assessments. This gap obscures failure modes that only manifest in situated contexts, such as misinterpretations that require iterative repair or outputs that are technically plausible but unusable for a specific user goal.

Methodology

The authors introduce MonitrLLM, an open-source, browser-based infrastructure designed to operationalize community-centered evaluation. The system architecture and pilot study methodology are as follows:

System Design

MonitrLLM consists of three components:

• Browser Extension: Integrates directly into the user's LLM workflow (specifically ChatGPT in the pilot). It activates via a toolbar icon, allowing users to submit an audit report without leaving their interface.
• Backend API: Built on Django, it receives form submissions, validates ChatGPT share links, retrieves full conversation transcripts, and stores linked evaluation records.
• Deployment Template: An open-source package allowing other communities to self-host or cloud-host the system, with configurable fields for task purpose, outcome assessment, and satisfaction ratings.

Data Collection Workflow:

1. Users install the extension and use LLMs naturally.
2. For interactions they wish to reflect on, users submit a five-field report:
   • Free-text interaction description.
   • Task purpose (user-defined intent).
   • Outcome assessment (user judgment of success/failure).
   • ChatGPT share link (to retrieve the full transcript).
   • Satisfaction rating (1–5 scale).
3. User identity is managed via a local UUID to ensure anonymity and minimize friction; no account registration is required.
4. Users retain full control to view or delete their submissions at any time.

Pilot Study

• Participants: 26 college students recruited from a U.S. university.
• Duration: Two weeks.
• Dataset: 206 initial reports were collected; 194 valid conversations (from 25 participants) formed the working dataset after excluding cases where share links were deactivated.
• Analysis Approach: A sequential mixed-methods design. Researchers first familiarized themselves with the data, inductively developed a codebook for request types, domains, and failure modes, and then systematically coded all conversations.
• Coding Dimensions:
  • Request Type: 9 categories (e.g., coding, writing, information seeking).
  • Topic Domain: 7 categories (e.g., academic coursework, personal use).
  • Failure Taxonomy: 6 categories derived from the combination of transcripts and user outcome notes (e.g., misinterpretation, interaction friction, factual errors).

Key Contributions

1. MonitrLLM Infrastructure: A reusable, open-source system that treats user intent and user-perceived outcomes as "first-class evaluative signals" rather than optional metadata. It structurally links full conversation transcripts with audit metadata.
2. Feasibility Demonstration: A pilot study proving that community-contributed metadata is viable and produces findings that are structurally absent from transcript-only collections or standard interface feedback.
3. Methodological Shift: A framework for "community-centered evaluation" where the community's tasks and definitions of success are the primary evaluative objects, rather than generic benchmark targets.

Key Results

The pilot analysis of 194 conversations revealed insights that would remain invisible without the linkage of transcripts, intent, and outcomes:

• The Satisfaction-Failure Gap: Despite a high average satisfaction rating of 4.19/5, the dataset revealed a 23.1% failure rate (45 of 194 conversations) based on user-reported outcomes. High aggregate satisfaction masked significant task failures.
• Multi-Turn Conversations as Failure Signals: Contrary to the assumption that more turns indicate deeper engagement, follow-up conversations failed at 2.5 times the rate of single-turn exchanges (30.0% vs. 12.0%). Failed interactions required significantly more user turns (mean 6.11) than successful ones (mean 3.12), indicating that extended interaction often signals difficulty and repair strategies rather than productive engagement.
• Failure Taxonomy: The most common failure types were:
  • Misinterpretation and Reframing (33.3% of failures): The model answers a different question than intended.
  • Dissatisfaction Unspecified (24.4%): Users report negative outcomes without a detectable specific mechanism in the transcript alone.
  • Interaction Friction/Non-convergence (22.2%): Repeated prompts fail to yield usable output.
• Domain-Specific Risks: Failure rates were highest in academic contexts (30.0% combined for coursework and research), particularly in coding/debugging (32.0%) and summarization (33.3%). Personal and everyday use showed a lower failure rate (21.7%).
• Transcript Limitations: Many failures (e.g., subtle factual errors, unverifiable citations, or "hallucinations" that users partially detected) were only legible when the transcript was read alongside the user's outcome note. In some cases, transcripts appeared productive while users were quietly managing unreliable model behavior.

Significance and Claims

The paper argues that the central contribution is infrastructural: evaluation can only surface what the surrounding system preserves as evidence. Current interfaces fail to preserve the link between what the user was trying to do, how they judged the outcome, and the interaction trajectory.

• Visibility of Hidden Costs: By pairing outcome assessments with transcript trajectories, MonitrLLM makes visible the "cost" of interaction (e.g., time spent troubleshooting) and specific failure modes (e.g., misinterpretation) that aggregate satisfaction scores conceal.
• Complementarity: The infrastructure does not replace benchmarks or red teaming but complements them. While benchmarks measure capabilities under controlled conditions and red teaming probes adversarial stress, MonitrLLM captures failures in ordinary, situated use where users are not trying to break the model but simply trying to complete their work.
• Community Agency: The system enables communities to define their own success criteria and failure taxonomies, shifting evaluation from expert-driven to community-led.
• Modest Scope: The authors explicitly frame the pilot as a feasibility study demonstrating what the infrastructure makes analytically visible, rather than making broad generalizable claims about all student LLM use. The significance lies in the possibility of this data linkage, not just the specific statistics of the pilot.

The paper concludes that as LLMs become embedded in consequential knowledge work, evaluation infrastructure must evolve to capture user intent and situated outcomes to provide a robust understanding of model performance in the wild.