Technical Summary: Context-as-a-Service (CaaS)

Problem Statement

The paper addresses the "cross-file documentation problem," a specific failure mode in LLM-assisted documentation workflows. While LLM agents can generate fluent and locally plausible documentation, their accuracy often degrades when claims depend on non-obvious dependency chains distributed across the codebase.

Standard LLM workflows typically rely on ordinary repository tools (file reads, keyword search, symbol navigation). These tools are effective when evidence is local or lexically obvious. However, they struggle when a documentation claim requires tracing semantic dependencies that are not immediately adjacent, such as:

• A method delegating to an internal scheduler.
• An equality check relying on a separate backing object.
• A tutorial step depending on framework lifecycle rules.
• A component assuming the existence of another component created by framework logic.

In these scenarios, the documentation may appear correct within its own file but be factually wrong or incomplete when viewed against the broader system behavior. The core challenge is not merely a lack of context tokens, but the agent's inability to identify which specific files, symbols, or tests constitute the relevant evidence chain to verify a claim.

Methodology: Context-as-a-Service (CaaS)

The authors propose Context-as-a-Service (CaaS), a retrieval layer designed to augment LLM agents without replacing their decision-making process regarding which files to inspect. CaaS acts as a tool-callable service that agents query to surface evidence before deciding which dependency chains to follow.

System Architecture

CaaS operates as a four-stage pipeline:

1. Source Ingestion: The system ingests three source types:

   • Source Code: Implementation files, public APIs, internal utilities, and tests.
   • API References: Structured documentation extracted from code comments and published reference pages.
   • Upstream Documentation: Third-party platform or engine documentation relevant to the SDK.
     A fetcher normalizes these into document records with metadata (source type, file path, timestamp) to support freshness checks.

2. Storage and Indexing: Documents are stored in raw and enriched forms. The enriched form is tokenized, indexed using BM25 for lexical matching, and embedded using DRAMA (Ma et al., 2025) for semantic search. This dual-indexing supports both exact symbol matching and behavioral/semantic queries.

3. Retrieval Interface: Agents query CaaS via tool calls. The system combines BM25 (lexical) and DRAMA (dense) results using Reciprocal Rank Fusion (RRF). The output is a ranked list of snippets with document identifiers and source metadata. Agents can issue partial or behavior-based queries (e.g., "deferred removal semantics") rather than just symbol names.

4. Review Layer: Retrieved snippets serve as candidate evidence. In the study, a lightweight review prompt labeled claims as consistent, contradicted, or incomplete. Findings were retained only if the evidence supported a concrete correction.

Evaluation Protocol

The authors evaluated CaaS in a retrospective study on a production SDK (~200 source files) using Claude Sonnet 4.6. Two workflows were compared:

• Baseline: An LLM agent using standard repository tools (file reads, keyword search, symbol navigation) without CaaS.
• +CaaS: The same agent and tools, augmented with CaaS retrieval as an additional tool.

The comparison isolated the value of the retrieval layer, ensuring that basic repository access was present in both conditions. Findings were retained only if they addressed meaningful gaps missed by the baseline and required context beyond the immediate task file.

Key Contributions

1. Retrieval Architecture: A composable, agent-callable retrieval system combining keyword and dense search over code, examples, tests, API references, and upstream documentation.
2. Case Studies: Two evaluations on a production SDK demonstrating that CaaS surfaced review findings (factual errors, missing prerequisites, implementation bugs) that baseline workflows with ordinary tools missed.
3. Problem Characterization: A taxonomy of documentation dependency chains (Single-file, Cross-file, Cross-system) and an evidence trail showing that "Cross-file" issues are the primary domain where retrieval adds value over standard LLM workflows.

Results

Case Study 1: API Reference Improvement

• Task: Review and improve documentation comments on public members of a core source file.
• Baseline: Identified 5 missing documentation entries. Did not flag existing comments as incorrect.
• +CaaS: Produced the same 5 fixes and surfaced 4 additional findings:
  • 2 Cross-file factual errors: A lookup method argument was misidentified (type vs. attribute identifier) due to registry behavior; a disposal method implied immediate removal but was actually deferred by a state-management layer.
  • 2 Underspecified comments: An equality method definition missed a backing object dependency; an ownership method comment was vague, missing filter logic details.

Case Study 2: Tutorial Generation and Validation

• Task: Generate and validate a step-by-step tutorial for an SDK interaction feature.
• Baseline: Validated 17 API claims; the tutorial passed all stages.
• +CaaS: Surfaced 4 additional findings:
  • 1 Executable bug: An incorrect resource URI scheme that would cause double-nesting (syntactically valid but semantically wrong).
  • 1 API-idiom improvement: Use of a verbose constructor when a utility module offered a shorter equivalent.
  • 2 Critical omissions: Missing required base class for lifecycle callbacks; missing collision-support component dependency for mesh geometry (which only auto-generates for primitives).

Efficiency Metrics

Across five runs per condition:

• Wall-clock Time: Reduced by 22% in Case Study 1 and 34% in Case Study 2.
• Input Tokens: Reduced in both cases (e.g., from 17.4K to 14.6K in Case 1), as retrieval returned compact, pre-ranked snippets.
• LLM Calls: Increased significantly (e.g., from 2.4 to 17.6 in Case 1) because the agent issued retrieval queries as tool calls.
• Prompt Cache Hit: Improved (e.g., 84% to 91% in Case 1).

Significance and Claims

The paper claims that cross-file documentation inconsistency is a practical failure mode for LLM-assisted documentation. The significance of CaaS lies not in generating more text, but in exposing contradictions and missing prerequisites whose evidence resides outside the immediate task context (in utility files, tests, framework docs, etc.).

The authors assert that:

• Retrieval is a practical complement to LLM workflows, helping agents discover relevant cross-file evidence before documentation is trusted.
• The value of retrieval concentrates on Cross-file and Cross-system issues, where evidence is not lexically adjacent to the claim. Single-file issues are already well-handled by standard LLMs.
• The approach does not replace agent reasoning but provides a scoped mechanism to surface evidence, reducing the "non-obvious dependency chain" problem.

Limitations

The authors note that this is a case study on a single production SDK, not a broad benchmark. Findings may not generalize to other languages, repositories, or agent designs. The evaluation relied on retained review findings rather than a controlled human-subject study. Additionally, the current retrieval operates at document/file granularity, whereas symbol-level indexing could further improve precision.