Context-Mediated Domain Adaptation in Multi-Agent Sensemaking Systems

1. Problem Statement

Current Large Language Model (LLM) and multi-agent systems operate on a unidirectional prompt-response paradigm. In these systems:

• Ephemeral Interactions: User interactions are transient. Once a user receives an AI-generated artifact and manually corrects errors (e.g., fixing terminology, restructuring arguments, or adjusting emphasis), those corrections are treated as endpoint fixes.
• Loss of Tacit Knowledge: Domain experts possess "tacit knowledge" that is difficult to articulate via explicit prompts. When they edit AI outputs, they reveal deep domain understanding, but this information is lost to the system.
• Cold Start Problem: Every new session or new user faces a "cold start," forcing experts to repeatedly re-specify domain constraints and preferences that they have already demonstrated through previous edits.
• Inefficiency: This lack of feedback loops forces experts to spend significant time (5–60 hours in some workflows) on manual screening and corrections, as the system fails to learn from their implicit specifications.

2. Methodology: Context-Mediated Domain Adaptation (CMDA)

The authors propose Context-Mediated Domain Adaptation (CMDA), a paradigm where user modifications to system-generated artifacts serve as implicit domain specifications that reshape future agent reasoning.

Core Concepts

• Bidirectional Semantic Links: The system maintains persistent links between AI-generated content and user modifications. It does not just store the final output but analyzes the delta (the difference) between the initial generation and the user-edited version.
• Bidirectional Domain-Adaptive Representation (BDAR): A structured artifact format that preserves generation context (prompts, parameters) and modification history (edit distances, change patterns). This creates a "delta-based representation" capturing the gap between system knowledge and user expertise.
• Adaptive Context Object (ACO): A persistent knowledge store that accumulates extracted domain knowledge. It is organized into three categories:
  1. Domain Terminology Evolution: Vocabulary preferences and specialized language usage (e.g., changing "chart" to "visualization").
  2. Methodological Refinements: Improvements to research approaches, analytical frameworks, and domain-specific practices.
  3. Conceptual Depth Changes: Theoretical nuances, conceptual relationships, and domain-specific considerations not present in initial generations.

System Architecture: Seedentia

The authors implemented a prototype called Seedentia, a web-based multi-agent framework built with:

• Frontend: Next.js 15 with React, featuring an AIContentWrapper component that supports three interaction modes.
• Backend: Python FastAPI with LangGraph for orchestrating multi-agent workflows.
• Persistence: PostgreSQL for storing bidirectional semantic links and knowledge extraction.

Three Interaction Modes:

1. Direct Manipulation: Inline text editing (e.g., changing a word). Captures fine-grained terminology and structural preferences.
2. Prompt-based Regeneration: Users provide natural language instructions to regenerate content. Captures higher-level conceptual goals and methodological requirements.
3. Context-based Generation: The system generates new artifacts purely based on accumulated implicit knowledge from previous users, without further user input.

The Adaptation Loop:

1. Capture: User edits are logged, and edit distances are calculated.
2. Extract: An LLM-based pipeline analyzes the differences between the original and edited text to extract implicit domain knowledge (categorized into the three ACO types).
3. Propagate: Extracted knowledge is injected into the system prompt for subsequent generations, enriching the context for future users and future papers.

3. Key Contributions

1. The CMDA Paradigm: A theoretical framework for bidirectional human-AI interaction where user modifications reshape multi-agent behavior through structured knowledge representations (BDAR and ACO).
2. Seedentia Prototype: A working implementation demonstrating three interaction modes and a comprehensive logging infrastructure that supports cross-participant knowledge accumulation.
3. Implicit Knowledge Extraction: A method for extracting actionable domain expertise from edit patterns (e.g., inferring that "research questions must specify target user expertise" based on consistent edits adding such qualifiers).
4. Evaluation Framework: A study design demonstrating how accumulated knowledge improves generation quality and reduces correction effort across sequential users.

4. Results

The authors conducted an exploratory evaluation with 5 domain experts in visualization literacy who sequentially processed three academic papers.

• Knowledge Extraction: The system successfully extracted 46 unique domain knowledge entries from user modifications.
  • Distribution: 56.5% were Conceptual Depth Changes, 21.7% Methodological Refinements, and 21.7% Domain Terminology Evolution.
  • Insight: Experts primarily expanded the system's conceptual understanding rather than just fixing surface-level errors.
• Quality Improvement:
  • Baseline Ratings: Initial generation quality ratings increased from an average of 2.67–3.0 for the first participant (P1) to 3.56–4.33 for the last participant (P5).
  • Editing Effort: While edit distance did not decrease monotonically (later participants sometimes made more edits to refine deeper concepts), the nature of edits shifted from corrective to exploratory.
• Cross-User Transfer:
  • P3, despite benefiting from 22 prior knowledge entries, contributed the most new knowledge (20 entries), suggesting that accumulated knowledge enables deeper expert refinement rather than replacing it.
  • P4 and P5 required significantly fewer edits and achieved higher satisfaction ratings, demonstrating the effectiveness of cross-participant knowledge transfer.
• Efficiency: Session duration decreased for later participants (P1 avg: 6.3 min/paper vs. P4 avg: 5.6 min/paper), though individual variability was high.

5. Significance and Future Implications

• Bridging the Gap: CMDA effectively bridges the gap between generic LLM capabilities and the need for specialized domain expertise, allowing systems to "learn" from how experts correct them rather than just what they ask.
• Scalability: The approach moves beyond individual personalization to collective intelligence, where one expert's corrections improve the system for all subsequent users in the same domain.
• Engineering Impact: The paper provides reusable architectural patterns (BDAR, ACO, LangGraph workflows) for building adaptive, self-improving AI systems that do not require retraining (fine-tuning) but instead rely on context injection.
• Limitations & Future Work: The current study is limited to visualization literacy and a small sample size. Future work aims to scale to larger communities, handle non-text artifacts, implement knowledge "aging" mechanisms to prevent outdated patterns, and make the accumulated knowledge transparent and editable to end-users.

In summary, this paper demonstrates that by treating user edits as a continuous feedback loop for domain adaptation, multi-agent systems can evolve from generic tools into specialized, expert-aligned collaborators.