💻 computer science

Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways

This paper introduces GateScope, a black-box measurement framework that audits commercial LLM API gateways across content, conversation, billing, and latency dimensions, revealing significant discrepancies between advertised policies and actual behaviors such as silent model substitutions, pricing deviations, and inconsistent performance.

Original authors: Guanjie Lin, Yinxin Wan, Shichao Pei, Ting Xu, Kuai Xu, Guoliang Xue

Published 2026-04-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Guanjie Lin, Yinxin Wan, Shichao Pei, Ting Xu, Kuai Xu, Guoliang Xue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are ordering a fancy meal from a high-end restaurant. You ask the waiter for the "Chef's Special: Truffle Risotto." You pay the price listed on the menu for that specific dish.

Now, imagine the waiter is actually a middleman (a "Gateway"). They take your order, go into the kitchen, and come back with a plate. But here's the catch: You can't see the kitchen. You don't know if the waiter actually went to the head chef, or if they just grabbed a frozen meal from the back, or if they swapped your truffle risotto for a cheaper pasta dish but kept the extra change in their pocket.

This paper, titled "Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways," is about a team of researchers who decided to investigate these "waiters" (called LLM Gateways) to see if they are honest.

The Problem: The "Black Box" Waiters

Large Language Models (LLMs) like GPT-4 or Claude are like the master chefs. They are powerful and expensive. But using them directly can be complicated for companies. So, third-party companies built Gateways. These gateways act as a single front door. You give them one password, and they promise to let you talk to any chef you want.

The problem? These gateways are black boxes.

The Menu Lie: You ask for "GPT-5," but the gateway might secretly send your request to a cheaper, dumber model (like GPT-3.5) to save money.
The Silent Cut: You ask for a long story, but the gateway might silently chop off the end of the story without telling you.
The Fake Bill: You might be charged for 1,000 words when you only used 800, or charged for "cached" (pre-written) words that you didn't actually need to pay for.
The Slow Service: Sometimes the food comes in 2 seconds, sometimes in 2 minutes, and no one knows why.

The Solution: "GateScope" (The Food Critic)

The researchers built a tool called GateScope. Think of GateScope as a super-smart, invisible food critic who orders from these gateways thousands of times to see what's really happening. They don't have a key to the kitchen; they just act like a normal customer and analyze the results.

They checked the gateways in four specific ways:

1. The "Taste Test" (Response Content)

The Analogy: If you ask a master chef to solve a complex math problem, they will do it a certain way. If you ask a short-order cook, they might get the answer wrong or explain it differently.
What they did: GateScope asked the gateways tricky questions (like math puzzles or geography riddles) that require deep thinking. They analyzed the style of the answer.
The Findings: Some gateways were honest. Others were "fraudsters." When asked for a top-tier model, some gateways secretly served a cheaper model. The "taste" of the answer gave them away.

2. The "Memory Game" (Multi-turn Conversation)

The Analogy: Imagine you are having a long dinner conversation. You tell the waiter your favorite color is "Blue." Then you talk about the weather for 20 minutes. Finally, you ask, "What is my favorite color?" A good waiter remembers. A bad one forgets or changes the subject.
What they did: GateScope had long conversations with the gateways, planting secret clues early on and checking if the gateway remembered them later.
The Findings: Some gateways were great at remembering. Others were terrible. Some would "forget" your preferences because they secretly switched to a model with a smaller memory, or they cut off the beginning of your conversation to save space.

3. The "Receipt Check" (Billing Accuracy)

The Analogy: You order a burger and fries. The receipt says you ate a steak and a lobster.
What they did: They counted exactly how many "tokens" (words/pieces of information) they sent and received, then compared it to the bill the gateway sent them.
The Findings: Most gateways were fair. But a few were overcharging. Some charged for words that were already "cached" (pre-stored), or they just added extra fees that didn't match their public price list.

4. The "Stopwatch Test" (Latency)

The Analogy: You order a coffee. Sometimes it arrives in 10 seconds. Sometimes it takes 5 minutes. If it's a busy coffee shop, that's normal. But if the wait time jumps wildly between 10 seconds and 5 minutes for the same order, something is weird. Maybe they are sending your order to different kitchens.
What they did: They timed how long it took to get answers.
The Findings: Some gateways were very consistent. Others were chaotic. High variation in time often meant the gateway was switching between different models or struggling with its own internal traffic.

The Big Reveal

After testing 10 different commercial gateways, the researchers found that the "waiters" were not always trustworthy.

Model Switching: Some gateways frequently swapped the expensive model you paid for with a cheaper one.
Silent Truncation: Some gateways cut off your conversation history without telling you.
Billing Gaps: Some gateways overcharged by significant margins (one was off by nearly 63%!).
Unstable Speed: Some gateways were unpredictable, making it hard to rely on them for time-sensitive tasks.

Why This Matters

This paper is a wake-up call. As more businesses and people rely on these "all-in-one" gateways to access AI, they are trusting these middlemen with their data, their money, and the quality of their work.

GateScope proves that just because a gateway says it's using a specific model, doesn't mean it actually is. It's like ordering a steak and getting a burger, but the waiter insists it's a "premium burger." The researchers have made their tools public so that anyone can check if their "waiter" is honest.

In short: If you are paying for a premium AI service through a third-party gateway, you might not be getting what you paid for. This paper gives you the tools to check the receipt, taste the food, and make sure the kitchen isn't playing tricks on you.

1. Problem Statement

Third-party Large Language Model (LLM) API gateways have emerged as unified access points that aggregate models from multiple vendors (e.g., OpenAI, Anthropic, Google) into a single interface. While they offer benefits like simplified billing and load balancing, they operate as black-box intermediaries. This opacity creates significant risks for users:

Model Substitution/Downgrading: Gateways may silently route requests to cheaper or less capable models than the one advertised.
Silent Truncation: Gateways may impose hidden context window limits, discarding parts of the conversation history or truncating responses without notifying the user.
Billing Inaccuracies: Discrepancies between reported token usage and actual charges, including improper handling of cached tokens or hidden fees.
Latency Instability: Unpredictable response times caused by internal routing, model switching, or resource contention.

Existing research focuses on open-source model fingerprinting or first-party vendor APIs, leaving a gap in auditing the behavioral transparency of commercial third-party gateways.

2. Methodology: GateScope

The authors introduce GateScope, a lightweight, black-box measurement framework designed to audit LLM gateways across four critical dimensions. It operates entirely through public APIs without requiring internal access.

A. Response Content Analysis (Model Fingerprinting)

Goal: Detect if the gateway is serving the claimed model or a substitute.
Technique: Instead of relying on unstable free-text generation, GateScope uses a Discrete Concept Composition (DCC) approach.
- Prompt Design: It employs a probe suite covering four domains: AIME (math reasoning), GPQA (deep reasoning), Factual Recall (knowledge cutoff), and Geographic Inference (multi-hop reasoning).
- Structured Output: Prompts force models to output a strict JSON schema containing knowledge_path (reasoning steps) and final_answer.
- Signature Extraction: The framework extracts a $d$ -dimensional behavioral signature vector based on features like answer correctness, reasoning depth, step length, style (e.g., LaTeX usage), and parsing success.
- Classification: Lightweight XGBoost classifiers (one-vs-rest) are trained to distinguish specific model signatures from others.

B. Multi-turn Conversation Performance

Goal: Evaluate context retention and detect silent model switching or truncation during long interactions.
Technique: A standardized 25-turn conversation template is used.
- Protocol: The conversation assigns an identity and preference, introduces distractors, updates the preference mid-dialog, and finally queries the model to recall both the original identity and the updated preference.
- Metrics:
  1. Memory Checkpoint Pass Rate: Accuracy of recalling information at turns 10, 24, and 25.
  2. System Fingerprint Stability: Count of distinct system_fingerprint values (changes indicate model switching).
  3. Cached Token Usage Rate: Ratio of cached tokens to total input (switching models often resets cache hits).

C. Billing Accuracy

Goal: Verify if charges align with public pricing and reported token usage.
Technique: The framework calculates the Expected Cost ( $C_{expected}$ $C_{e x p ec t e d}$ ) based on the gateway's published rates and the actual token counts (input, output, and cached) reported in the API response.
- Formula: $C_{expected} = (n_{in} - n_{cached}) \times p_{in} + n_{cached} \times p_{cached} + n_{out} \times p_{out}$ .
- It compares this against the Actual Amount ( $C_{act}$ ) charged by the gateway to identify billing gaps.

D. Latency Characteristics

Goal: Measure stability and detect backend inconsistencies.
Technique: Repeated requests are sent under fixed conditions.
- Metric: The Coefficient of Variation (CV) of latency ( $\sigma_T / \mu_T$ ) is calculated. High CV or bimodal distributions suggest heterogeneous backend behaviors (e.g., frequent model switching or load balancing issues).

3. Key Contributions

Identification of Transparency Gaps: The paper highlights the lack of guarantees in LLM gateways regarding model identity, context preservation, and billing, motivating the need for client-side auditing.
GateScope Framework: A novel, lightweight system that audits gateways across content, conversation, billing, and latency dimensions using structured probing and behavioral fingerprinting.
Empirical Validation: The framework was validated on official vendor APIs (achieving high classification accuracy) and applied to 10 real-world commercial gateways, revealing significant discrepancies between advertised and actual behaviors.
Open Artifacts: The authors released the GateScope code and datasets on GitHub to facilitate further research.

4. Results

The evaluation of 10 commercial gateways revealed frequent misbehaviors:

Model Substitution: Several gateways showed low consistency in serving the claimed models. For example, when requesting gpt-5, some gateways were frequently identified as serving models outside the gpt-5 family. For gpt-4o, identification rates dropped to as low as 46.91% on certain platforms.
Degraded Memory Retention: Many gateways failed to retain context in long conversations. Some platforms showed 0% cache usage even when the baseline API showed ~40%, suggesting they were either not using the claimed model's caching mechanism or were switching models entirely.
Billing Deviations: While most gateways were accurate, outliers existed. One platform (o*ey) showed a 62.8% billing gap for gpt-4o, and another (a*ix) showed a 7.6% gap, despite similar token usage to others.
Latency Instability: Significant variation in latency was observed. Platform b*ie exhibited a Coefficient of Variation (CV) of 1.10 for math tasks (compared to 0.63 for the baseline), indicating highly unstable backend processing.

5. Significance

This work is significant because it shifts the focus from analyzing the models themselves to analyzing the infrastructure delivering them.

Trustworthy AI: It provides a mechanism for users to verify that they are getting what they pay for, ensuring that "premium" models are not being downgraded to save costs.
Economic Impact: By exposing billing inaccuracies and silent truncation, it protects enterprise users from financial loss and service degradation.
Methodological Advance: The use of structured JSON outputs and behavioral signatures offers a robust alternative to traditional text-based fingerprinting, which is often unreliable due to generation randomness.
Policy Implications: The findings suggest a need for stricter transparency regulations or industry standards for third-party LLM gateways to prevent "black-box" manipulation of model selection and pricing.