🤖 AI

Specification Portability Across LLM Development Agents: Cross-Agent Compatibility in Specification-Driven Software Migration

This paper demonstrates that specifications generated by one AI agent for Oracle-to-PostgreSQL migration often fail to port effectively to other agents, revealing significant agent-dependent degradation in implementation quality and highlighting the need for explicit strategies like retrieval-augmented ingestion to ensure cross-agent compatibility in software engineering workflows.

Original authors: Oleg Grynets, Oleksii Ilchuk, Dariia Zatulna, Vasyl Lyashkevych

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Oleg Grynets, Oleksii Ilchuk, Dariia Zatulna, Vasyl Lyashkevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of software creation, a new kind of worker has joined the team: the large language model. These are powerful computer programs trained on vast amounts of text and code, capable of reading a description of a task and writing the instructions a computer needs to perform it. As these tools become more common, developers are shifting from simply asking them to write code to giving them detailed blueprints, known as specifications. These specifications act as operational guides, telling the model exactly what to build, how it should behave, and what rules it must follow. This approach, called specification-driven development, promises to make software creation more reliable and structured. However, a critical question has emerged as teams begin to use multiple different models to build a single system: if one model writes a perfect blueprint, will a different model be able to read it and build the same thing? The assumption has been that a good plan is a good plan, regardless of who reads it, but the reality of how these machines interpret information is far more complex.

Researchers at EPAM Systems set out to test this assumption by treating software migration as a controlled experiment. They chose a specific, difficult task: moving database code from one system, Oracle, to another, PostgreSQL. These two systems speak similar languages but have different dialects, requiring precise translation of logic, data types, and functions. The team first established a baseline by having a single model generate a specification and then immediately use that same specification to write the new code. This worked reasonably well; out of over a thousand source files, the system successfully regenerated more than 600, and nearly 400 of those new scripts ran correctly in the target environment. This proved that the method of using a middle-step specification was viable. But the real test came when they introduced a second, different model into the mix.

The researchers created a scenario where one model, such as Amazon Kiro, would write a specification, and then a completely different model, like Google Gemini or GitHub Copilot, would be handed that document to generate the code. They wanted to see if the second model could understand the first model's plan without losing quality. The results were striking and surprising. The size of the specification turned out to be irrelevant to the outcome. One model produced a massive, detailed document with nearly 1,600 lines of text, while another produced a concise version with only about 200 lines. Yet, the length of the document did not predict how well the code would work. In fact, the most significant finding was that the origin of the specification mattered immensely. When Google Gemini was given a specification written by Amazon Kiro, the quality of the resulting code collapsed. The new scripts failed to run, contained syntax errors, and looked nothing like the intended target. This failure was not a one-time glitch; the researchers repeated the experiment and saw the same dramatic drop in performance, confirming that the two models simply could not agree on how to interpret the same set of instructions.

This incompatibility was not universal, however, which added a layer of nuance to the discovery. While Gemini struggled deeply with Kiro's specifications, GitHub Copilot handled the same foreign documents much better, sometimes even performing as well with them as it did with its own. This suggested that the problem was not that foreign plans were inherently bad, but that different models have different ways of reading and understanding text. To address this, the team tested several ways to help the models bridge the gap. They tried rewriting the foreign specifications into a new format that the receiving model might prefer, and they tried compressing the text to make it shorter. Rewriting helped Gemini significantly, bringing its performance back up to a usable level, but compressing the text offered no real benefit. The most promising strategy involved a technique called retrieval-augmented generation. Instead of feeding the entire specification to the model at once, the researchers gave the model a tool to search through the document and pull out only the specific parts it needed for the current task. This approach did not win on every single measure, but it was the only method that provided a consistently strong balance of performance for both the struggling and the successful models.

The study concludes that in a world where software is built by a team of different artificial intelligence agents, a specification cannot be treated as a neutral, universal document. A plan written by one agent is not automatically a valid instruction set for another. The effectiveness of the code depends heavily on the specific relationship between the model that wrote the plan and the model that is building the software. If a team swaps out one agent for another, they cannot simply assume the existing blueprints will work; they may need to adapt the language of the plan or change how the new agent accesses the information. The research suggests that the future of multi-agent software engineering will require a new focus on how specifications are structured and delivered, ensuring that the knowledge contained in a plan can actually be understood by the machine tasked with building it.

Technical Summary: Specification Portability Across LLM Development Agents

1. Problem Statement

The paper addresses a critical interoperability gap in emerging multi-agent software engineering (SDE) workflows. As Large Language Models (LLMs) transition from prompt-based assistants to autonomous agents that generate code from specifications, a new question arises: Is a specification created by one LLM agent environment reliably consumable by another?

Current research focuses on single-agent performance, context retrieval, or requirements clarification. However, it remains empirically unclear whether a specification generated by Agent A (e.g., Amazon Kiro) retains its operational effectiveness when consumed by Agent B (e.g., Google Gemini) to produce target code. The authors hypothesize that specifications are not "agent-neutral" artifacts; their effectiveness may degrade significantly depending on the origin of the specification and the ingestion strategy used by the implementation agent.

2. Methodology

The study employs a controlled, two-stage experimental design using Oracle-to-PostgreSQL database migration as the transformation task. This domain was chosen because it requires preserving complex application semantics while translating dialect-specific constructs (procedural logic, triggers, data types).

Stage 1: Specification-Driven Migration Pipeline

Task: A specification-first pipeline was evaluated on 1,006 PL/SQL source files.
Process: Source code was decomposed into database objects. An agent generated an intermediate textual specification, which was then used to derive PostgreSQL implementation.
Outcome: 623 files were successfully regenerated, and 380 of those executed successfully in PostgreSQL 16. This established the feasibility of the approach but did not yet test cross-agent transfer.

Stage 2: Cross-Agent Compatibility Experiments

Dataset: 1,802 Oracle scripts with corresponding PostgreSQL implementations.
Agents Evaluated: Amazon Kiro, Google Gemini (via Antigravity), GitHub Copilot (via Spec Kit), Claude Code, and Cursor.
Experimental Conditions:
1. Native Baseline: Each agent generated its own specification and implementation.
2. Cross-Agent Transfer: Specifications generated by one agent were fed to another (e.g., Kiro spec $\to$ Gemini implementation).
3. Adaptation Strategies: To mitigate potential degradation, three strategies were tested on foreign specifications:
  - Rewriting: The implementation agent rewrites the foreign spec into its preferred format before coding.
  - Compression: Summarizing the foreign spec to reduce length.
  - Retrieval-Augmented Generation (RAG): The agent accesses the foreign spec via retrieval tools rather than ingesting the full text directly.

Evaluation Metrics

The study utilized a multidimensional metric set to distinguish lexical, syntactic, structural, and operational quality:

Token F1: Token-level similarity.
Exact Match: Percentage of outputs identical to the target.
SQL Syntax Validity: Validity of generated SQL (using sqlglot).
AST Exact Match & Mean Similarity: Structural similarity of Abstract Syntax Trees.
Immediate Runnability: Percentage of scripts executing without manual modification (noted as a lower bound due to test environment constraints).

3. Key Results

A. Specification Size is Not a Predictor of Quality

Contrary to the assumption that more detailed specifications yield better code, the study found no correlation between specification length and implementation quality.

Kiro generated the largest specifications (~1,597 lines).
Gemini generated significantly smaller specifications (~193 lines).
Despite the size difference, the larger Kiro specifications did not universally produce superior code across all agents or metrics.

B. Asymmetric and Agent-Dependent Degradation

Cross-agent transfer produced highly variable results, confirming that specification portability is directional ( $C_{i \to j} \neq C_{j \to i}$ ).

Critical Failure Case: When Gemini directly consumed a Kiro-origin specification, performance collapsed.
- Token F1 dropped to 0.035 (vs. 0.73 native).
- SQL Syntax Validity dropped to 2.33% (vs. 45.54% native).
- AST Mean Similarity dropped to 0.015.
- This failure pattern was replicated in a second independent run, ruling out random nondeterminism.
Stability in Other Pairs: Other combinations (e.g., Copilot consuming Kiro specs) remained relatively stable or showed mixed improvements, indicating that degradation is not a universal law but specific to agent-specification pairings.

C. Effectiveness of Adaptation Strategies

Rewriting: Substantially improved Gemini's performance when consuming Kiro specs, suggesting that representation format (not just content) is a primary source of incompatibility. This strategy was less effective for Copilot.
Compression: Did not provide a universal benefit; simple length reduction did not solve incompatibility.
Retrieval-Augmented Ingestion (RAG): While RAG did not dominate every single metric, it was the only strategy that appeared on the Pareto frontiers for both Gemini and Copilot when handling foreign specifications. This identifies RAG as the most robust common trade-off strategy among the tested alternatives.

4. Key Contributions

Empirical Evidence of Non-Portability: The paper provides the first controlled empirical evidence that specifications generated in one LLM environment are not automatically effective in another, challenging the assumption of agent-neutral intermediate artifacts.
Multidimensional Evaluation Framework: The study introduces a comprehensive evaluation suite (Token F1, AST similarity, runnability) specifically for cross-agent specification transfer, moving beyond simple code generation benchmarks.
Adaptation Strategy Analysis: It systematically evaluates rewriting, compression, and RAG, identifying that representation adaptation (rewriting) and selective access (RAG) are more effective than simple content reduction.
Formal Model Proposal: The authors propose a model where implementation quality $Q_{code}$ is a function of the specification origin ( $A_g$ ), the implementation agent ( $A_i$ ), the ingestion strategy ( $I_s$ ), and context ( $C$ ), distinguishing between knowledge preservation and operational interpretation preservation.

5. Significance and Claims

The paper modestly claims that its findings are empirical and context-specific, restricted to the agents, datasets, and metrics evaluated. It does not claim that foreign specifications are universally inferior, but rather that their effectiveness is agent-dependent.

Primary Significance:

Workflow Implications: In heterogeneous multi-agent SDD workflows, specifications should not be treated as static, interchangeable artifacts. Transferring a specification between agents may require explicit adaptation (e.g., rewriting or RAG) to prevent implementation degradation.
Design Guidance: The results motivate the development of structured specification representations (separating knowledge, representation, dependencies, and granularity) and agent-specific views of specifications.
Future Direction: The study suggests that retrieval-based access to specifications is a robust strategy for unknown agent pairings, but further research is needed on dependency-aware retrieval and structured specification models to fully solve the portability problem.

The authors conclude that "specification portability" is an open problem requiring explicit consideration of how information is structured, transferred, and interpreted across different LLM development environments.