💻 computer science

Context Matters: Evaluating Context Strategies for Automated ADR Generation Using LLMs

This paper demonstrates that for automated Architecture Decision Record (ADR) generation using Large Language Models, strategically selecting a small window of recent historical context (typically 3–5 records) is more effective for balancing quality and efficiency than relying on model scale or complex retrieval-based strategies.

Original authors: Aviral Gupta, Rudra Dhar, Daniel Feitosa, Karthik Vaidhyanathan

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aviral Gupta, Rudra Dhar, Daniel Feitosa, Karthik Vaidhyanathan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Forgotten Notebook"

Imagine you are building a massive, complex Lego castle. Every time you make a big decision—like "We need a red tower here because the blue one is too wobbly"—you write it down in a notebook. This notebook is called an Architecture Decision Record (ADR).

In the real world of software, these notebooks are crucial. They tell future builders why the castle looks the way it does. But here's the problem: writing these notes is boring and takes time. Architects often skip them, or they write them poorly, leaving future teams confused. It's like trying to fix a car without knowing what the previous mechanic did.

The Solution: The AI Assistant

The researchers asked: Can an AI (a Large Language Model or LLM) write these notes for us?

They knew AI is good at writing, but they suspected the AI might be "hallucinating" (making things up) if it didn't know the history of the project. So, they ran a massive experiment to see how much history the AI needs to read before it can write a good note.

The Experiment: The "Time Travel" Test

The researchers gathered 750 real-world software projects and over 4,500 real notes. They then asked the AI to write a new note for a project, but they gave the AI different "time capsules" of history to read.

They tested five different strategies:

The Amnesiac (No Context): The AI is given the title of the new decision but nothing else. It has to guess the rest.
- Result: It wrote long, rambling, generic stories that didn't fit the specific project. Like a tourist guessing the rules of a local game without watching anyone play.
The Hoarder (All History): The AI reads every single note ever written for that project, from day one to yesterday.
- Result: It did a good job, but it was slow, expensive, and sometimes got confused by old, irrelevant notes (like reading a 10-year-old diary entry while trying to decide what to wear today).
The Historian (First-K): The AI only reads the first few notes (the foundational decisions).
- Result: It understood the basics but missed recent changes. It was like knowing the foundation of a house but not knowing that the roof was replaced last week.
The Gossip (Last-K): The AI only reads the last 3 to 5 notes.
- Result: This was the winner. It produced the best notes, almost as good as reading everything, but much faster and cheaper. It realized that what happened yesterday matters more than what happened ten years ago.
The Detective (RAFG): The AI uses a smart search to find notes that are conceptually similar, even if they are old.
- Result: This was great for weird, complex situations where a decision from 2 years ago is still relevant today. But for most normal days, it was overkill and didn't beat the "Gossip" strategy.

The Big Discoveries (The "Aha!" Moments)

1. Context is King, Not Model Size
You might think you need the most expensive, super-smart AI to write good notes. The study found that even a smaller, cheaper AI could write amazing notes if it was given the right context (the last 3-5 notes).

Analogy: It's not about having a genius student; it's about giving them the right textbook. A smart student with no book will fail, but a regular student with the right book will ace the test.

2. The "Sweet Spot" is Small
You don't need to read the whole library to write a good summary. Reading the last 3 to 5 entries was usually enough.

Analogy: If you want to know the plot of a TV show, you don't need to watch every episode from Season 1. You just need to watch the last few episodes to understand what's happening right now.

3. The "External Link" Trap
The researchers noticed that when human-written notes said, "See our Wiki for details," the AI struggled. The AI tried to write a full explanation, but the "score" was low because it didn't match the short note that just said "See Wiki."

Analogy: If a human says, "Ask the librarian," and the AI writes a whole paragraph about the book, the AI gets a bad grade for not matching the short note, even though the AI actually did the hard work of explaining it! This shows that bad human habits (hiding info in links) make it hard to measure if the AI is doing a good job.

What Should We Do? (The Takeaway)

If you are building a tool to help software teams write these notes automatically:

Don't feed the AI the whole history. It's too much noise.
Give it the last 3 to 5 notes. This is the "Goldilocks" zone—just right.
Don't worry about buying the most expensive AI. A smaller, local AI works just fine if you give it the right context.
Fix your own notes first. If your team hides information in external links, the AI can't help you. You need to write self-contained notes.

In a Nutshell

The paper proves that how you talk to the AI matters more than how "smart" the AI is. By giving the AI a small, recent slice of history (the last few decisions), we can automate the boring work of documentation without needing supercomputers or reading the entire history of the project. It's about giving the AI the right context, not just a bigger brain.

1. Problem Statement

Architecture Decision Records (ADRs) are essential for preserving the rationale behind software design decisions, yet their adoption is inconsistent due to the high "authoring overhead." Architects often neglect documentation because the immediate cost of writing ADRs outweighs the long-term benefits.

While Large Language Models (LLMs) offer a potential solution to automate ADR drafting, existing research treats ADR generation as an isolated text-generation task. This approach ignores a fundamental property of software architecture: path dependency. Architectural decisions are rarely made in a vacuum; they are constrained by historical decisions, established technology stacks, and previous trade-offs.

The core problem addressed is: How should historical ADRs be presented as context to an LLM to maximize the quality of generated ADRs? Specifically, the authors investigate whether providing all history, recent history, foundational history, or semantically retrieved history yields better results, and whether complex retrieval mechanisms are necessary compared to simple chronological strategies.

2. Methodology

The study employs a rigorous empirical approach involving a large-scale dataset and multiple LLM families.

Dataset Construction

Source: The authors curated a dataset from 750 open-source repositories (originally identified in a study by Buchgeher et al.), resulting in over 4,500 validated, sequential ADRs.
Preprocessing: They performed manual filtering to ensure chronological integrity, removing structural noise, empty stubs, and non-architectural documentation. This created a "next-step" generation dataset where the model predicts the $N$ -th ADR based on the title of the $N$ -th ADR and the history of $1$ to $N-1$ .

Experimental Design

The study evaluates five context strategies across four distinct LLM families (ranging from lightweight local models to proprietary SOTA models):

Baseline (No Context): The model receives only the target ADR title.
All-N (Full History): All preceding ADRs are provided.
First-K: Only the first $K$ (foundational) ADRs are provided.
Last-K: Only the most recent $K$ ADRs are provided.
RAFG (Retrieval-Augmented Few-shot Generation): Uses semantic similarity (via SentenceTransformer embeddings) to dynamically retrieve the top- $K$ most relevant ADRs from the entire history, regardless of chronological order.

Note: $K$ was empirically determined to be 3 or 5, representing the point of diminishing returns.

Evaluation Metrics

Generated ADRs were compared against human-written ground truth using:

Automated Metrics: BERTScore (semantic similarity), BLEU (n-gram precision), ROUGE (recall/overlap), and METEOR (synonym/stemming alignment).
Token Usage: To measure efficiency and verbosity.
Manual Review: A subset of outputs was reviewed to assess rationale quality and formatting.

3. Key Contributions

Curated Sequential Corpus: The creation of a large-scale, chronologically validated ADR corpus (750 repos, 4,500+ records) specifically designed for "next-step" generation experiments.
Context Engineering Framework: A systematic comparison of context strategies, moving beyond simple model benchmarking to analyze how information retrieval impacts architectural reasoning.
Empirical Validation of Recency: Evidence that a small window of recent history is sufficient for high-quality generation, challenging the assumption that "more context is always better."
Practical Guidelines: Concrete recommendations for tool builders regarding context selection, model selection, and documentation hygiene.

4. Key Results

RQ1: Impact of Decision History

Context is Critical: Providing historical context significantly improves generation quality compared to the no-context baseline.
Verbosity Reduction: Without context, LLMs generate verbose, generic text (avg. ~~1123 tokens). With context, outputs become concise and rationale-heavy (avg. ~630–730 tokens), closely matching human ground truth (~~526 tokens).
Quality Alignment: Context-aware strategies align the generated rationale with human expectations, whereas the baseline fails to capture architectural constraints.

RQ2: Completeness vs. Efficiency (Selective Strategies)

Diminishing Returns: Providing the entire history (All-N) offers negligible performance gains over a small, strategic subset.
Optimal Window: A window of 3 to 5 recent records (Last-K) provides the best balance of quality and efficiency.
- Example: For Gemini-2.5-Pro, Last_K(3) achieved a BERTScore F1 of 0.8440, nearly identical to All_N (0.8421) and significantly better than First_K (0.8401).
Foundational vs. Recent: First-K (foundational decisions) consistently underperformed Last-K, suggesting that recent architectural shifts are more predictive of current decisions than early project foundations.

RQ3: Chronological vs. Retrieval (RAFG)

Parity in Aggregate: RAFG (semantic retrieval) performs comparably to Last-K in aggregate metrics.
Specialized Utility: While Last-K is sufficient for linear workflows, RAFG offers a specialized advantage for cross-cutting concerns or non-sequential dependencies (e.g., a security decision made months ago relevant to a current database change).
Trade-off: The marginal gain of RAFG over Last-K often does not justify the increased implementation complexity and retrieval latency, unless the system specifically deals with complex, non-linear architectural drift.

Model Scale Findings

Context > Scale: Smaller, locally deployable models (e.g., Gemma-3-4B) with appropriate context strategies produced ADRs comparable in quality to massive proprietary models (e.g., Gemini-2.5-Pro).
Implication: The bottleneck for ADR automation is context engineering, not model parameter count.

5. Significance and Implications

For Practitioners

Default Strategy: Adopt Last-K (3–5 records) as the default context strategy. It is simple, low-latency, and highly effective.
Model Selection: Organizations can use smaller, locally hosted models to ensure data privacy and reduce costs without sacrificing ADR quality, provided the context is well-engineered.
Documentation Hygiene: The study highlights that "external content dependency" (linking to wikis/external docs instead of writing self-contained rationale) degrades LLM performance. Teams must maintain self-contained ADRs to enable effective automation.

For Researchers

Shift in Focus: Future research should prioritize context selection strategies and hybrid retrieval (combining recency with semantic search) over simply scaling model parameters.
Evaluation Challenges: Current metrics penalize LLMs for generating self-contained text when ground-truth ADRs rely on implicit external knowledge. Future datasets need to be curated to be self-contained to ensure fair evaluation.
Architectural Locality: The concept of "temporal locality" (recent decisions matter most) and "conceptual locality" (semantic relevance matters for cross-cutting concerns) offers a new lens for studying architectural evolution.

Conclusion

The paper demonstrates that context engineering is the dominant factor in automating ADR generation. A simple strategy of using the 3–5 most recent ADRs yields near-optimal results, making complex retrieval systems unnecessary for most linear workflows. This finding democratizes ADR automation, allowing organizations to use cost-effective, local models while maintaining high-quality architectural documentation.