Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads
This paper presents "Local-Splitter," an open-source framework that systematically evaluates seven tactics for reducing cloud LLM token usage in coding agents by leveraging a local triage model, finding that combining local routing with prompt compression yields the highest savings (45–79%) for edit-heavy and explanation-heavy workloads while demonstrating that the optimal strategy is workload-dependent.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you own a high-end, super-expensive consulting firm (the Cloud LLM). Your clients are developers who ask your consultants thousands of questions a day. Some questions are deep, complex, and require a PhD-level expert. But many are simple: "What does this file do?" or "Fix this typo."
Right now, your firm is losing money because you're sending every single question, even the simple ones, to your expensive senior consultants. It's like hiring a Nobel Prize-winning physicist to tell you how to tie your shoelaces.
This paper introduces Local-Splitter, a smart "receptionist" (a small, cheap computer program running on your own laptop) that stands in front of your expensive consultants. Its job is to filter, sort, and simplify requests so your expensive team only does the heavy lifting when absolutely necessary.
Here is how the paper breaks down the 7 Tactics this receptionist uses, explained with everyday analogies:
The 7 Tactics (The Receptionist's Toolkit)
Local Routing (The "Bouncer"):
- The Idea: Before a question reaches the expensive consultant, the receptionist checks if it's "trivial." If it's a simple "What is this?" question, the receptionist answers it themselves.
- Analogy: A bouncer at a club who stops people who just want to use the bathroom from entering the VIP lounge. Only the VIPs (complex questions) get in.
- Result: Saves a huge amount of money because 30–70% of questions never reach the expensive team.
Prompt Compression (The "Summarizer"):
- The Idea: Developers often send huge, rambling emails with 10,000 words of context. The receptionist reads it, cuts out the fluff, and sends a tight 400-word summary to the consultant.
- Analogy: Instead of reading a 300-page novel to a friend, you give them a 2-page "CliffsNotes" version that still has all the plot points.
- Result: The consultant charges less because they read fewer words.
Semantic Caching (The "Memory Bank"):
- The Idea: If a user asks, "How do I fix this bug?" and then asks it again five minutes later (or if someone else asks the same thing), the receptionist just pulls the answer from a notebook instead of calling the consultant again.
- Analogy: A librarian who remembers you asked for the same book yesterday and just hands it to you without calling the author.
Local Drafting with Cloud Review (The "Intern + Editor"):
- The Idea: The receptionist (a smart intern) writes a first draft of the answer. Then, the expensive consultant just reads it and says, "Looks good, just fix this one sentence."
- Analogy: Instead of a senior editor writing an article from scratch, they just proofread a draft written by a junior writer. It's much faster and cheaper to edit than to write.
- Note: This only works well if the intern is good enough to get 90% of it right. If the draft is terrible, the editor has to rewrite everything, which costs more.
Minimal-Diff Edits (The "Scissors"):
- The Idea: If a user wants to change one line of code in a 10,000-line file, the receptionist cuts out the whole file and only sends the 50 lines around the change.
- Analogy: If you want to fix a typo in a book, you don't mail the whole book to the printer. You just mail the one page with the typo.
Structured Intent Extraction (The "Form Filler"):
- The Idea: Users often chat in a rambling way ("Hey, could you maybe help me with..."). The receptionist strips away the polite chatter and turns it into a strict form: "Task: Fix Bug. File: main.py."
- Analogy: Turning a 5-minute phone conversation into a 3-line text message that gets straight to the point.
Batching & Caching (The "Group Order"):
- The Idea: If a user fires off 5 quick questions in a row, the receptionist waits a split second, groups them into one big request, and sends them together.
- Analogy: Instead of ordering 5 separate pizzas and paying 5 delivery fees, you order one big pizza with 5 toppings to save on the delivery charge.
The Big Discovery: "One Size Does Not Fit All"
The most important finding of the paper is that you shouldn't turn all these tactics on at once.
Think of it like a toolbox.
- If you are fixing a leaky faucet (a "Chat" or "Edit" workload), you just need a wrench (Routing) and a rag (Compression). If you try to use a sledgehammer (Drafting) too, you'll break the pipe and make a mess.
- If you are demolishing a wall (a "RAG" or complex research workload), you need the sledgehammer (Drafting) because the job is so big that having the intern do the heavy lifting first saves time.
The Paper's Verdict:
- For most coding tasks: Use Routing (T1) and Compression (T2). This combination saves 45% to 79% of the cost.
- For complex research tasks: You can add the Drafting tactic (T4) to save even more, but only if the answers are long.
- Don't turn everything on: If you use the "Drafting" tactic on simple tasks, it actually increases your bill because the "review" process adds extra words that the consultant has to read.
The Bottom Line
The authors built a free, open-source "wrapper" (Local-Splitter) that sits between your coding agent and the expensive AI cloud. By using a small, free AI model on your own computer to do the heavy lifting of sorting and summarizing, companies can slash their AI bills by nearly half without making the answers any worse.
It's the difference between paying a celebrity chef to cook every single meal for your family versus having a smart sous-chef handle the prep work and only calling the celebrity chef for the final plating.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.