🤖 AI

LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

LLM-Rosetta is an open-source, hub-and-spoke intermediate representation framework that enables lossless, bidirectional translation between major LLM API providers, effectively solving the $O(N^2)$ adapter problem by mapping diverse syntactic formats to a unified semantic core with minimal overhead.

Original authors: Peng Ding

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peng Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a traveler visiting four different countries: OpenAI, Anthropic, Google, and OpenAI's new "Responses" service.

In each country, the locals speak a slightly different dialect of the same language. They all want to talk about "messages," "tools," and "thinking," but they write their sentences differently.

In OpenAI, you might say, "Here is a list of messages."
In Anthropic, you might say, "Here is a list of content blocks."
In Google, you might say, "Here are parts of a conversation."

If you want to build a travel app that works in all four countries, you currently face a nightmare. To make your app work everywhere, you have to build a specific translator for every single pair of countries.

OpenAI $\leftrightarrow$ Anthropic
OpenAI $\leftrightarrow$ Google
Anthropic $\leftrightarrow$ Google
...and so on.

If there are 100 countries, you need thousands of translators. This is the $O(N^2)$ problem: the work grows explosively as you add more countries.

Enter LLM-Rosetta: The Universal "Hub"

The authors of this paper, led by Peng Ding, realized that despite the different dialects, everyone is actually saying the same things. They all talk about:

Who is speaking? (System, User, Assistant)
What is being said? (Text, Images, Audio)
What tools are being used? (Calculators, Search engines)
How should we think? (Reasoning steps)

They built LLM-Rosetta, a tool that acts like a universal currency exchange or a central train station.

Instead of building a direct road between every pair of countries, they built a central hub (the "Intermediate Representation" or IR).

The Spoke: You take your specific request (e.g., from OpenAI) and translate it once into the Universal Language (the Hub).
The Hub: The Hub understands the core meaning perfectly. It doesn't care about the weird formatting of the original country.
The Spoke: The Hub then translates that Universal Language into the specific dialect of the destination country (e.g., Google).

Now, if you want to add a 5th country (like a new AI provider), you don't need to rewrite all your old translators. You just build one new translator from that country to the Hub. The math changes from a messy explosion ( $N^2$ ) to a simple addition ( $N$ ).

How It Works (The "Ops" Kitchen)

The paper describes a clever way to build these translators called "Ops-Composition."

Imagine a kitchen where you need to cook a meal for four different cultures. Instead of one chef trying to remember four different recipes at once, you have four specialized stations:

The Content Station: Handles the food (text, images, files).
The Message Station: Handles the order of the meal (who speaks first).
The Tool Station: Handles the utensils (calculators, code).
The Config Station: Handles the temperature and timing (how fast to think, how creative to be).

When you need to translate from OpenAI to Google, you just swap out the "OpenAI Chef" for the "Google Chef" at each station. The rest of the kitchen stays the same. This makes the system modular, easy to fix, and easy to test.

The "Live Stream" Challenge

One of the hardest parts of talking to AI is that it doesn't reply all at once; it "streams" words one by one, like a live video feed. Different providers send these "chunks" of data in different ways.

OpenAI might send a "start" signal, then "text," then "end."
Google might send the whole sentence so far, then the next part.

LLM-Rosetta acts like a live subtitle translator. It watches the incoming stream, normalizes it into the "Universal Language" (10 standard event types), and then re-streams it in the format the destination expects. It keeps track of the "state" so that if a tool call starts in one chunk and finishes in another, the translation doesn't get confused.

Why Is This a Big Deal?

It's Lossless: You can send a message from OpenAI to the Hub and back to OpenAI, and it will look exactly the same (down to the tiny metadata details), just like translating a book to French and back to English without losing a single plot point.
It's Fast: The translation happens in less than 100 microseconds (that's 0.0001 seconds). It's so fast that it doesn't slow down your app at all.
It's Already Working: This isn't just theory. It's currently being used at Argonne National Laboratory to help researchers switch between different AI models without rewriting their code.

The Bottom Line

LLM-Rosetta is the Rosetta Stone for AI. It realized that while AI companies speak different "dialects," they all share the same "grammar." By creating a universal middleman, it frees developers from being locked into one company. You can now build an app that talks to OpenAI today, and if you want to switch to Google tomorrow, you just flip a switch. The app doesn't need to be rebuilt; it just needs a new translator for the Hub.

1. Problem Statement

The Large Language Model (LLM) ecosystem is highly fragmented, with major providers (OpenAI, Anthropic, Google, etc.) exposing proprietary API formats.

The $O(N^2)$ Integration Burden: To connect $N$ providers, a naïve approach requires $N(N-1)/2$ bilateral adapters. As providers evolve their APIs (e.g., adding reasoning traces, changing streaming schemas), maintaining these adapters becomes combinatorially expensive and brittle.
Syntactic Divergence vs. Semantic Core: While API schemas differ significantly in syntax (field naming, nesting depth, content encoding), the paper observes that they share a common semantic core (messages, content parts, tool calls, reasoning, and generation controls).
Limitations of Existing Solutions:
- Application Frameworks (e.g., LangChain): Abstract at the application level but do not provide a reusable, format-level translation layer for raw API payloads.
- SDK Wrappers (e.g., LiteLLM): Often map all providers to a single "lingua franca" (usually OpenAI Chat Completions). This approach loses provider-specific features (e.g., Anthropic's cache control, Google's grounding metadata) and lacks true bidirectional fidelity.

2. Methodology: LLM-Rosetta Architecture

LLM-Rosetta proposes a Hub-and-Spoke architecture using a typed Intermediate Representation (IR) to decouple providers, reducing complexity from $O(N^2)$ to $O(N)$ .

A. The Intermediate Representation (IR)

The IR is a provider-neutral data model designed to capture the semantic union of supported APIs. It consists of:

9 Content Part Types: TextPart, ImagePart, AudioPart, FilePart, ToolCallPart, ToolResultPart, ReasoningPart, RefusalPart, CitationPart.
10 Stream Event Types: Normalized events for streaming (e.g., stream_start, content_block_start, text_delta, tool_call_delta, finish, usage).
Core Structures:
- Messages: Role-tagged containers (System, User, Assistant, Tool).
- Tools: Definitions and choice configurations.
- Config: Generation parameters (temperature, top-p, reasoning budgets).
- Requests/Responses: Standardized wrappers for payloads.

B. Ops-Composition Converter Architecture

Instead of monolithic converters, LLM-Rosetta uses a modular design where each provider adapter is assembled from four orthogonal Ops modules:

ContentOps: Handles atomic content conversion (e.g., base64 encoding, tool call serialization).
MessageOps: Handles conversation structure, role mapping, and system prompt extraction.
ToolOps: Manages tool definitions and selection logic.
ConfigOps: Maps generation parameters and caching settings.

This separation allows providers to reuse modules (e.g., OpenAI Responses reuses OpenAI Chat's content logic) and isolates syntactic quirks from semantic logic.

C. Streaming and State Management

Streaming is a critical challenge due to differing Server-Sent Event (SSE) schemas. LLM-Rosetta implements a stateful StreamContext that:

Normalizes disparate event streams (e.g., Anthropic's explicit block lifecycle vs. OpenAI's implicit deltas) into the 10 IR event types.
Handles out-of-order events (e.g., Google emitting finish reasons before final content deltas) by buffering and reordering.
Supports chunk-level translation with sub-millisecond overhead.

D. Metadata Modes

The system supports two conversion modes:

Preserve Mode: Retains provider-specific metadata (e.g., cache_control, thought_signature) in a provider_metadata field, enabling lossless round-trips ( $A \to IR \to A$ ).
Strip Mode: Discards provider-specific metadata to produce clean, provider-neutral IR for cross-provider forwarding ( $A \to IR \to B$ ).

3. Key Contributions

Empirical Characterization: Demonstrated that LLM API divergence is primarily syntactic, allowing a compact IR (9 content types, 10 stream types) to faithfully represent the major providers.
Bidirectional Translation: Unlike single-direction wrappers, LLM-Rosetta supports full bidirectional conversion ( $Provider \leftrightarrow IR \leftrightarrow Provider$ ) for both requests and responses.
Modular Extensibility: The Ops-composition pattern allows new providers to be added by implementing only the specific Ops modules without modifying the core IR or existing converters.
Production Deployment: The framework is deployed in production at Argonne National Laboratory as the translation layer for the ARGO-PROXY gateway.

4. Evaluation Results

The system was evaluated using 1,364 tests (including the Open Responses compliance suite) and production benchmarks.

Round-Trip Fidelity: Achieved lossless field-level equality in Preserve mode across all four supported providers (OpenAI Chat, OpenAI Responses, Anthropic, Google GenAI).
Streaming Correctness: Maintained 100% event-sequence accuracy, correctly handling complex scenarios like tool call accumulation and out-of-order provider events.
Cross-Provider Translation: Successfully translated semantic content (roles, tools, multi-modal data) between all six provider pairs. Provider-specific features without equivalents (e.g., Anthropic cache control in Google) were safely dropped with warnings.
Performance Overhead:
- Latency: Median conversion overhead is < 100 µs (21–77 µs depending on payload complexity).
- Comparison: While slightly slower than LiteLLM's single-pass approach for complex multi-turn conversations (2.2x overhead), it remains negligible compared to network latency (50–500 ms) and model inference time. Crucially, it offers bidirectionality and provider neutrality which LiteLLM lacks.

5. Significance

LLM-Rosetta addresses a critical infrastructure gap in the LLM ecosystem. By treating API translation as a semantic mapping problem rather than a syntactic one, it enables:

Portability: Applications can switch or bridge providers without rewriting glue code.
Interoperability: Enables gateways and frameworks to route traffic between incompatible APIs transparently.
Future-Proofing: The modular design ensures that as LLM APIs evolve (e.g., new content types or reasoning features), the system can adapt incrementally without a complete rewrite.

The paper concludes that while the "combinatorial surface" of API variations is complex, the shared semantic core makes a hub-and-spoke IR a viable and efficient solution for building a unified, multi-provider LLM infrastructure.