Technical Summary: AIANO – Enhancing Information Retrieval with AI-Augmented Annotation

Problem Statement

The proliferation of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems has created an urgent demand for high-quality, curated datasets for information retrieval (IR) tasks. However, the creation of these datasets is currently hindered by the limitations of existing "off-the-shelf" annotation tools. Standard tools often fail to address the specific complexities of IR annotation, which requires synthesizing information across multiple documents, constructing contextual rationales, and ensuring comprehensive coverage. Consequently, dataset creation in specialized domains remains prohibitively challenging, time-consuming, and expensive. Furthermore, incomplete or noisy annotations resulting from inefficient workflows can lead to significant performance degradation in the retrieval models trained on such data. While fully automated labeling using LLMs has shown promise, it lacks the necessary human oversight to ensure trustworthiness and domain-specific accuracy, necessitating a collaborative approach that balances automation with human expertise.

Methodology

To address these challenges, the authors developed AIANO (AI-Augmented anNOtation), a specialized annotation tool designed to integrate human expertise with LLM capabilities within a semi-automatic workflow.

System Architecture and Design

AIANO models annotation tasks as configurable AIANO Blocks, which operate in three distinct modes to facilitate varying levels of human-AI collaboration:

1. Plain Mode: The AI performs no operations; the annotator manually writes all content (e.g., free-text comments).
2. AI Solo Mode: The AI automatically generates content based on pre-defined system prompts, which the annotator then reviews and refines (e.g., auto-generating boilerplate questions).
3. Human-AI Collaborative Mode: The AI synthesizes outputs from multiple sources (existing annotations, user-defined fields, highlighted passages, and system prompts). The annotator retains full control to accept, modify, or override these suggestions.

The system supports flexible input/output schemas defined via JSON, requiring only document and subject IDs as mandatory fields, allowing adaptation to diverse document types without programmatic configuration. The architecture utilizes a containerized microservices approach with a React 19 frontend, FastAPI backend, PostgreSQL database, and Docker deployment, supporting any LLM provider compliant with OpenAI API standards (including local deployments like vLLM).

User Study Design

The authors evaluated AIANO through a controlled within-subject user study involving 15 participants (graduate students, researchers, developers, medical doctors, and regulatory specialists).

• Comparison: AIANO was compared against Label Studio, a widely used general-purpose annotation tool.
• Task: Participants created question-answering datasets for RAG evaluation using 60 short German general knowledge documents. Tasks involved identifying relevant documents, highlighting passages, and formulating answers for both single-document and multi-document retrieval scenarios.
• Configuration: In AIANO, participants used a Question Block (Plain Mode) and an Answer Block (Human-AI Collaborative Mode) powered by Meta's Llama 70B model. Label Studio was configured without native interactive AI collaboration or full-text search capabilities.
• Metrics: Evaluation included objective measures (task completion time, retrieval precision, recall, and F1 score) and subjective measures (NASA-TLX cognitive load assessment and a custom usability questionnaire).

Key Contributions

The paper outlines three primary contributions:

1. AIANO Tool: The introduction of a specialized annotation tool specifically designed for information retrieval tasks, distinguishing it from general-purpose platforms.
2. Design Principles: The articulation of design principles that enable efficient LLM-human collaboration in semi-automatic workflows, specifically through the block-based architecture and collaborative modes.
3. Systematic Evaluation: A controlled within-subject study demonstrating the effectiveness and usability of AIANO compared to a baseline tool, providing empirical evidence of its impact on speed, cognitive load, and data quality.

Results

The study yielded significant quantitative and qualitative improvements with AIANO over the baseline:

• Efficiency: Participants completed annotation tasks nearly twice as fast with AIANO. The median task completion time decreased from 10.0 minutes (Label Studio) to 6.0 minutes (AIANO).
• Cognitive Load: AIANO significantly reduced the overall workload. NASA-TLX scores showed substantial reductions in mental demand (15.0 vs. 35.0), physical demand (5.0 vs. 20.0), effort (15.0 vs. 35.0), and frustration (10.0 vs. 50.0).
• Usability: Participants rated AIANO significantly higher across all usability dimensions, including interface intuitiveness, ease of use, and navigation. The composite usability score was 4.25 for AIANO compared to 2.375 for Label Studio.
• Retrieval Performance: Datasets created with AIANO demonstrated superior retrieval metrics.
  • Precision: 0.889 (AIANO) vs. 0.867 (Label Studio)
  • Recall: 0.883 (AIANO) vs. 0.783 (Label Studio)
  • F1-Score: 0.860 (AIANO) vs. 0.787 (Label Studio)
  • This represents an average improvement of 8.2% across the three metrics, with recall showing the most significant gain (+12.8%).

Qualitative feedback indicated that 93.3% of participants found the AI assistance useful, and 86.7% found the search functionality useful, with 80% valuing the combination of both for efficiency.

Significance and Claims

The authors claim that AIANO successfully addresses the fundamental workflow inefficiencies inherent in document-intensive IR annotation. By strategically integrating LLM assistance while preserving human oversight, the tool accelerates dataset creation without compromising quality. The results suggest that AIANO's approach not only reduces the time and cognitive burden on annotators but also enhances the quality of the resulting datasets, leading to more robust retrieval systems.

The paper positions AIANO as a middle ground between traditional tools lacking intelligent assistance and overly generic platforms without domain-specific affordances. The authors emphasize that their work demonstrates how dataset creation for IR tasks can be significantly enhanced by embedding LLMs directly into the annotation workflow as interactive collaborators. However, they modestly acknowledge limitations, noting that the study was limited to 15 participants, focused on German-language short documents, and compared against a single baseline, suggesting that broader validation across diverse domains and languages is necessary to establish generalizability.