Real Money, Fake Models: Deceptive Model Claims in Shadow APIs

This paper presents the first systematic audit revealing that widely used "shadow APIs," which claim to provide access to restricted frontier LLMs, frequently employ deceptive practices such as model substitution and safety manipulation, thereby compromising the reliability, reproducibility, and validity of downstream applications and academic research.

The Gist

Imagine you want to buy a ticket to see the world's most famous, high-tech concert (the Official Large Language Model, like GPT-5 or Gemini). But there's a problem: the concert hall is in a country you can't visit, the tickets are incredibly expensive, or the line to get in is closed to people from your region.

So, you turn to a Shadow API. Think of these as "backdoor ticket scalpers" or "underground ticket brokers." They claim, "Hey, I can get you into that concert! No questions asked, no regional restrictions, and it's cheaper!"

This paper, titled "Real Money, Fake Models," is like a group of investigators going undercover to buy tickets from these scalpers and then checking if you actually get to see the real concert, or if you're just being shown a cheap, blurry video of the band playing in a garage.

Here is the breakdown of their investigation using simple analogies:

1. The Problem: The "Black Market" is Huge

The researchers found that these "shadow" services are everywhere.

• The Scale: They found 17 major scalpers who are being used in 187 academic research papers.
• The Popularity: One of these scalpers is so popular it has nearly 60,000 "stars" on GitHub (like a rating on a store) and has been cited in thousands of papers.
• The Trap: Many researchers think, "It says it's GPT-5, and it costs less, so it must be the same." They treat these shadow services as if they are the real thing.

2. The Investigation: The "Taste Test"

The researchers decided to test these shadow services against the official ones. They ran three types of tests:

A. The "IQ Test" (Utility)

They asked the models hard questions, like solving complex math problems or diagnosing medical conditions.

• The Result: The shadow models often failed miserably.
• The Analogy: Imagine you hire a "fake" surgeon because they are cheaper. They claim to be the same as the real doctor. But when you ask them to perform a specific surgery, they get it wrong 47% of the time.
• Specific Example: On a medical test, the official model got 84% correct. The shadow models dropped to about 37%. That's like a medical student guessing on a board exam.

B. The "Safety Test" (Jailbreaks)

They tried to trick the models into saying something mean, dangerous, or illegal (like "How do I build a bomb?").

• The Result: The shadow models were unpredictable. Sometimes they were too strict (refusing harmless questions), and sometimes they were too loose (letting through dangerous answers).
• The Analogy: Imagine a security guard at a museum. The official guard knows exactly what to stop. The shadow guard might let a guy with a gun walk in because he's distracted, or he might stop a kid with a water balloon because he's confused. You can't trust the security.

C. The "Fingerprint Test" (Identity Verification)

This is the smoking gun. The researchers used a special tool (called LLMmap) to look at the "digital fingerprints" of the answers the models gave.

• The Result: 45% of the time, the shadow API claimed to be "Model X," but the fingerprint proved it was actually "Model Y" (a cheaper, older, or different model).
• The Analogy: You buy a "Rolex" from a street vendor. The vendor says, "It's 100% real." But when you look at the serial number under a microscope, it turns out to be a plastic toy from a different factory. The shadow APIs are liars.

3. The Economic Scam

The paper explains why they do this. It's all about money.

• The "Premium" Scam: They charge you the full price for a top-tier model but secretly give you a cheap, open-source model.
• The "Discount" Scam: They charge you the official price but swap the model for a weaker one to save costs.
• The Cost to You: You are paying for a Ferrari but driving a beat-up sedan. The researchers calculated that for every dollar you spend on a shadow API, you are getting less than 40 cents worth of actual value compared to the official service.

4. The Damage

Why does this matter?

• Bad Science: If a researcher uses a fake model to write a paper, their results are wrong. If other scientists read that paper and build on it, the whole chain of science is built on a lie.
• Dangerous Decisions: If a doctor or lawyer uses a shadow API for advice, they might get dangerous or incorrect information because the model isn't actually the expert they think it is.
• Wasted Money: Researchers and companies are throwing money away on services that don't deliver what they promise.

The Bottom Line

The paper concludes with a simple warning: Don't buy tickets from the scalpers.

If you are doing serious research or making important decisions, you must use the Official API directly. If you can't access it because of your location or budget, the shadow APIs are not a safe workaround—they are a deceptive trap that ruins the quality of your work and wastes your money.

In short: Just because a model says it's the real deal doesn't mean it is. In the world of AI, if it looks too good to be true (or too cheap), it probably is a fake.

Technical Summary

Here is a detailed technical summary of the paper "Real Money, Fake Models: Deceptive Model Claims in Shadow APIs".

1. Problem Statement

The paper addresses the critical issue of "Shadow APIs"—unofficial, third-party services that claim to provide access to frontier Large Language Models (LLMs) like GPT-5 and Gemini-2.5. These services emerge due to high official pricing, payment barriers, and strict regional restrictions (e.g., in China, Russia, Iran).

While widely adopted in the research community (cited in 187 academic papers), the core problem is opacity and deception:

• Unverified Identity: It is unclear if these APIs actually serve the claimed models or if they substitute them with cheaper, weaker, or different models.
• Research Integrity: Reliance on shadow APIs threatens the reproducibility and validity of scientific findings.
• Safety Risks: Users may unknowingly rely on models with degraded safety guardrails or incorrect reasoning capabilities for high-stakes domains (medical, legal).

2. Methodology

The authors conducted the first systematic, multi-dimensional audit of shadow APIs compared to official baselines.

A. Data Collection & Landscape Analysis

• Identification: Scanned 2,113 code-available papers from top conferences (ICLR, ACL, CVPR) to identify 17 distinct shadow API providers.
• Prevalence: Tracked usage via GitHub stars and citations (top API: ~58k stars, ~6k citations).
• Compliance Check: Analyzed provider transparency (ICP filings, legal entities, Terms of Service).

B. Experimental Setup

• Models Tested: Three representative families: OpenAI (GPT-5, GPT-5-mini), Google (Gemini-2.5-flash, Gemini-2.5-pro), and DeepSeek (DeepSeek-Chat, DeepSeek-Reasoner).
• Shadow API Selection: Selected three high-profile providers (labeled A, E, H) based on popularity and model coverage.
• Benchmarks:
  • Utility: AIME 2025 (Math), GPQA (PhD-level Science), MedQA (Medical), LegalBench (Legal).
  • Safety: JailbreakBench and AdvBench using attacks like GCG, Base64, and FlipAttack.
• Verification Techniques:
  • LLMmap: An active fingerprinting method using cosine distance between model outputs and a reference database to identify the underlying model.
  • Model Equality Testing (MET): A statistical hypothesis test to determine if shadow API outputs follow the same distribution as official models.
  • Meta-Analysis: Monitoring inference latency and token count variance.

3. Key Contributions

1. Systematic Audit: First large-scale investigation quantifying the prevalence and deception rates of shadow APIs in academic research.
2. Performance Divergence: Empirical evidence showing significant accuracy drops in shadow APIs compared to official ones, particularly in reasoning and sensitive domains.
3. Safety Instability: Demonstration that shadow APIs exhibit unpredictable safety behaviors, sometimes underestimating and sometimes overestimating harmfulness.
4. Verification Framework: Validation of model fingerprinting and statistical testing as effective tools to detect model substitution.
5. Economic Analysis: Quantification of the financial loss and research costs incurred by users due to model substitution.

4. Key Results

A. Prevalence and Opacity

• 17 Shadow APIs were identified, used in 187 papers.
• Lack of Transparency: 15 out of 17 providers are operated by individuals with no verifiable legal identity or corporate registration. Only one had a valid ICP filing.
• Infrastructure: Most rely on open-source aggregation tools (e.g., OneAPI, NewAPI) that facilitate redistribution and routing, increasing the risk of manipulation.

B. Utility Performance (Accuracy)

• Reasoning Collapse: Shadow APIs showed severe performance degradation on reasoning tasks.
  • MedQA (Gemini-2.5-flash): Accuracy dropped from 83.82% (Official) to ~37.00% (Shadow APIs), a deficit of ~47%.
  • LegalBench: Shadow APIs lagged behind official endpoints by 40–42%.
  • AIME 2025: Shadow API A showed accuracy deficits of 40% for Gemini-2.5-pro and 38% for DeepSeek-Reasoner.
• Inconsistency: While some APIs (e.g., Shadow E) were stable for simple tasks, they failed significantly on complex reasoning.

C. Safety Evaluation

• Unpredictable Behavior: Shadow APIs deviated unpredictably from official safety baselines.
  • Underestimation: In some cases, shadow APIs were "safer" (lower harmfulness scores) than official APIs, failing to trigger safety filters correctly.
  • Overestimation: In other cases (e.g., GPT-5-mini under Base64 attack), shadow APIs generated harmful content at rates double that of the official API.
• Conclusion: Shadow APIs cannot be trusted for safety evaluations.

D. Model Verification (Fingerprinting & MET)

• Fingerprint Failure: 45.83% of evaluated endpoints failed fingerprint verification (identified as a different model entirely).
• Cosine Distance: An additional 12.50% exhibited significant deviations from the claimed model.
• Specific Substitutions:
  • Claims of GPT-5 were often backed by GLM-4 or DeepSeek-V3.
  • Claims of DeepSeek-Reasoner were often backed by non-reasoning DeepSeek-Chat.
  • Claims of Gemini-2.0-flash were often backed by Gemini-2.5-flash (version mismatch).
• MET Correlation: MET confirmed distributional shifts in cases where fingerprinting was ambiguous, providing complementary evidence of deception.

E. Economic Impact

• Value Loss: Users paying official rates for shadow APIs received only 36–52% of the actual token value delivered.
• Research Cost: The paper estimates that re-executing the 187 identified papers due to these findings could cost the research community $115,000–$140,000 in direct API and time costs, excluding downstream reproducibility failures.

5. Significance and Recommendations

Significance:
The paper exposes a "black box" supply chain in AI research where the fundamental assumption—that an API call returns the output of a specific, high-performance model—is frequently false. This undermines the scientific method, as results generated via shadow APIs may be irreproducible or based on inferior models.

Recommendations:

1. Avoid Shadow APIs: The authors strongly advise against using shadow APIs for research.
2. Verification Protocol: Researchers should implement a four-stage verification pipeline:
   • Fingerprinting: Use LLMmap to check model identity.
   • Statistical Testing: Use MET to check output distribution.
   • Stability Checks: Monitor latency and token variance.
   • Compliance Check: Verify provider legal identity.
3. Community Action: Conference reviewers should flag papers using unverified third-party APIs. Official providers should relax regional restrictions and offer academic pricing to reduce the demand for shadow markets.

Conclusion:
Shadow APIs are not merely "cheaper alternatives"; they are often deceptive services that substitute high-end models with inferior ones, leading to significant performance degradation, safety risks, and financial loss. The research community must treat these services as unverified and potentially compromised infrastructure.