🤖 machine learning

Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation

The paper proposes AReS, an efficient adaptation framework that primes a local encoder via a single API interaction and then performs glass-box reprogramming locally, thereby achieving superior performance on modern closed-box models while eliminating nearly all subsequent API costs compared to traditional zeroth-order optimization methods.

Original authors: Yunbei Zhang, Chengyi Cai, Feng Liu, Jihun Hamm

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunbei Zhang, Chengyi Cai, Feng Liu, Jihun Hamm

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-genius chef (the "Service Model," like GPT-4o or a massive AI) who lives in a locked kitchen. You can't see their recipes, you can't touch their ingredients, and you can't change how they cook. The only way to interact with them is to send them a photo of a dish and ask, "What is this?" They send back an answer.

Now, imagine you want this chef to become an expert at identifying rare flowers (a specific task they weren't originally trained for).

The Old Way: The "Noisy Shout" Strategy (ZOO)

Previously, researchers tried to teach this locked-away chef by shouting random noises at the kitchen door.

They would send a picture of a flower, then send the same picture with a tiny bit of digital "static" or noise added, then send it again with different noise.
They would listen to the chef's answers to guess, "Oh, when I added this noise, the chef changed their mind. So, this noise must be important!"
The Problem: This is incredibly expensive (you have to pay for every shout), slow, and frustrating.
The New Reality: Modern chefs (like GPT-4o) have become so smart and robust that they just ignore the noise. They look at the flower, see right through the static, and give the same answer. The "shouting" strategy stops working, but you've already spent a fortune trying.

The New Way: AReS (Prime Once, Then Reprogram Locally)

The authors of this paper propose a clever, two-step shortcut called AReS. Instead of trying to force the locked chef to learn, they use the chef to train a local apprentice in your own kitchen.

Step 1: The "Prime" (The One-Time Lesson)

You send a batch of flower photos to the super-chef just once.

You ask the chef, "What do you think this is?"
You don't try to change the chef. Instead, you take the chef's answers and use them to teach a local apprentice (a smaller, open-source AI model you own).
You train your apprentice to mimic the chef's thinking. This is the "Priming" stage. It's like the chef giving the apprentice a crash course in flower identification.
Crucial Point: You only talk to the expensive chef once. After this, you never need to call them again.

Step 2: The "Reprogram" (The Local Practice)

Now, you have a local apprentice who has "listened" to the chef's wisdom.

You take your apprentice into your own kitchen (your local computer).
Because you can see the apprentice's brain (it's "open-box"), you can easily tweak their brain using standard, fast, and free methods to get them perfect at identifying flowers.
You can do this as many times as you want, with zero cost, because you aren't calling the expensive chef anymore.

Why This is a Game-Changer

It's Cheaper: The old way cost thousands of dollars in API fees. AReS cuts that cost by 99.99%. You only pay for the one-time lesson.
It Works on Smart Chefs: Because AReS doesn't rely on "shouting noise" at the chef, it works perfectly even on the most robust, modern AI models (like GPT-4o) that ignore noise.
It's Offline: Once the apprentice is trained, you can take them to a cabin in the woods with no internet. They still know how to identify flowers because they learned from the chef before you left.
It's Smarter: Surprisingly, the local apprentice often ends up better at the specific task than the chef was, because the apprentice learned the chef's general wisdom and then specialized specifically for your needs without the chef's "baggage."

The Analogy in a Nutshell

Old Method: Trying to teach a genius by throwing pebbles at their window and guessing what they learned based on how they flinched. (Expensive, slow, and the genius ignores the pebbles).
AReS: Asking the genius to give a lecture to your student one time. Then, you take your student home, practice with them for free, and turn them into an expert.

The Result: You get a highly accurate flower identifier, you save a fortune, and you don't need an internet connection to use it.

1. Problem Statement

The paper addresses the challenge of adapting Closed-Box Model-as-a-Service (MaaS) models (e.g., GPT-4o, CLIP APIs, proprietary vision models) to specific downstream tasks without access to their internal parameters or gradients.

Current State: The standard approach is Model Reprogramming (MR) using Zeroth-Order Optimization (ZOO). Methods like BAR and BlackVIP modify input images with learnable "visual prompts" and estimate gradients via random perturbations (querying the API thousands of times).
Key Limitations:
1. High Cost & Latency: ZOO requires massive numbers of API calls during both training and inference, leading to prohibitive financial costs and latency.
2. Instability: Gradient estimation via random perturbations is noisy and often leads to slow or unstable convergence.
3. Ineffectiveness on Modern Models: The authors observe that modern, robust models (e.g., GPT-4o) are often invariant to the input perturbations ZOO relies on. Consequently, ZOO methods fail to improve performance over zero-shot baselines on these models, rendering the high cost unjustified.

2. Methodology: AReS (Alternative efficient Reprogramming approach for Service models)

AReS proposes a paradigm shift from continuous closed-box optimization to a two-stage "Prime Once, then Reprogram Locally" strategy.

Stage 1: Priming (Single-Pass Interaction)

Goal: To align a local, pre-trained encoder ( $F_L$ ) with the output distribution of the closed-box service model ( $F_S$ ) using a single round of API interaction.
Process:
- For each training image $x_i$ , query the service API once to get prediction probabilities $p_S(x_i)$ .
- Use these probabilities as "soft labels" to train a lightweight, trainable linear layer on top of the frozen local encoder $F_L$ .
- Loss Function: The training minimizes the Kullback-Leibler (KL) divergence (or cross-entropy) between the local model's output and the service model's output.
- Key Feature: This stage does not require the service model and the target task to share the same label space (e.g., it can prime a local model using ImageNet outputs to prepare it for a Flower classification task).
Outcome: The local model is "primed" to mimic the service model's behavior on the target data distribution.

Stage 2: Efficient Local Reprogramming (Glass-Box)

Goal: Learn the optimal visual prompt for the specific downstream task.
Process:
- Since the local model is now accessible (glass-box), the authors perform gradient-based optimization (First-Order Optimization) to learn the visual prompt $P$ .
- This is done entirely on the local machine using the primed local encoder.
Outcome: The model learns a robust visual prompt without any further API calls.

Inference

Cost-Free: All subsequent inference is performed locally using the primed encoder and the learned prompt. Zero API calls are required during deployment.

3. Theoretical Foundation

The paper provides a theoretical analysis bounding the performance of the local model relative to the service model.

$\epsilon$ -Faithful Priming: The authors define priming as "faithful" if the $L_1$ difference between the service model's logits and the local model's logits is bounded by a small $\epsilon$ .
Performance Bound: They prove that if priming is faithful ( $\epsilon$ is small), the risk (error) of the optimally reprogrammed local model ( $R_L$ ) is bounded by the risk of the optimally reprogrammed service model ( $R_S$ ) plus $\epsilon$ :
$R_L(D^T, P^*) - \epsilon \leq R_S(D^T, Q^*) \leq R_L(D^T, P^*)$
Implication: This theoretically justifies shifting the optimization burden from the expensive, unstable closed-box API to the efficient, stable local model, provided the initial priming is effective.

4. Key Contributions

Identification of a Critical Flaw: The authors demonstrate that ZOO-based methods fail on modern, robust APIs (like GPT-4o) because these models ignore the input noise ZOO relies on.
AReS Framework: A novel, theoretically grounded framework that decouples the adaptation process into a one-time "priming" step and a local "reprogramming" step.
Disjoint Label Space Handling: Unlike traditional Knowledge Distillation, AReS's priming stage works even when the service model's label space (e.g., ImageNet) is disjoint from the target task (e.g., Flowers), making it applicable to standard Vision Models (VMs) as well as Vision-Language Models (VLMs).
Cost-Free Inference: The method completely eliminates API dependency after the initial training phase.

5. Experimental Results

The authors evaluated AReS on 10 diverse datasets (Flowers102, EuroSAT, etc.) using various service models (CLIP, ViT, ResNet, GPT-4o, LLaVA, Clarifai).

Performance Gains:
- GPT-4o: AReS achieved a +27.8% accuracy gain over the zero-shot baseline. In contrast, ZOO-based methods (BAR, BlackVIP) provided little to no improvement (e.g., +0.7%).
- General Benchmarks: Across 10 datasets, AReS outperformed state-of-the-art ZOO methods by +2.5% for VLMs and +15.6% for VMs.
- ImageNet (16-shot): AReS achieved 80.1% accuracy, surpassing the glass-box baseline and significantly outperforming ZOO methods which degraded performance.
Efficiency & Cost:
- API Calls: Reduced API calls by >99.99% (e.g., from ~ $10^8$ calls to ~ $10^3$ calls).
- Training Time: Reduced training time by >98% (e.g., from ~32 hours to <0.5 hours).
- Financial Cost: On the commercial Clarifai API, AReS cost $0.20 compared to $67.30 for BlackVIP for the same task.
Robustness: AReS maintained high performance even with partial API outputs (top-k predictions) and across different domain gaps (e.g., medical imaging).

6. Significance

Paradigm Shift: AReS moves the field away from the inefficient "query-heavy" ZOO paradigm, which is becoming obsolete for modern robust models, toward a "local-first" adaptation strategy.
Practical Viability: By enabling offline, cost-free inference, AReS makes adapting powerful proprietary models feasible for edge devices, remote operations, and cost-sensitive applications.
Synergy: The method demonstrates that a local model, when properly primed, can leverage the knowledge of a massive service model to achieve performance that sometimes exceeds even the glass-box reprogramming of the service model itself (due to synergistic effects between the two models).

In summary, AReS offers a robust, economical, and theoretically sound solution for adapting closed-box AI services, solving the critical issues of cost, latency, and effectiveness that plague current black-box adaptation methods.