💬 NLP

Bridging the Knowledge Void: Inference-time Acquisition of Unfamiliar Programming Languages for Coding Tasks

This paper proposes **ILA-agent**, a framework that enables Large Language Models to master unfamiliar programming languages during inference by using behavioral primitives to interactively explore documentation and execution environments, a method validated through a new benchmark called **Cangjie-bench**.

Original authors: Chen Shen, Wei Cheng, Jingyue Yang, Huan Zhang, Yuhan Wu, Wei Hu

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chen Shen, Wei Cheng, Jingyue Yang, Huan Zhang, Yuhan Wu, Wei Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a world-class chef who has spent your entire life cooking only Italian cuisine. You know every pasta shape, every sauce, and every technique by heart. You are a master.

Suddenly, you are dropped into a kitchen in Japan. You are asked to make a perfect, authentic Sushi platter.

Even though you are a "genius" in the kitchen, you are currently useless for this task. You don't know the names of the fish, you don't know how to handle the rice, and if you try to use your Italian tools (like a pasta roller) on seaweed, you’ll make a mess.

This paper is about teaching "Master Chefs" (Large Language Models like ChatGPT) how to learn a brand-new "cuisine" (a new programming language) on the fly, without having to go back to culinary school.

The Problem: The "Expert's Blind Spot"

Current AI models are like those Italian chefs. They are incredibly smart, but their knowledge is "frozen" in time based on what they studied in school (their training data).

If a new programming language is invented tomorrow (like a new recipe), the AI won't know it. Usually, to fix this, scientists try to "re-train" the AI, which is like sending the chef back to school for four years—it’s incredibly expensive, slow, and exhausting.

The Solution: The "ILA-Agent" (The Smart Apprentice)

Instead of re-training the AI, the researchers created a framework called ILA-agent.

Think of the ILA-agent not as a chef who knows everything, but as a brilliant apprentice who is given three specific tools to survive in a new kitchen:

The Cookbook (Exploration Primitives): Instead of guessing, the agent can look at the official manual. It doesn't just flip pages randomly; it has a "Table of Contents" tool to find the right chapter and a "Search" tool to look up specific ingredients.
The Test Stove (Verification Primitives): This is the most important part. If the apprentice tries a new technique, they don't just assume it works. They actually "cook" a small sample and see if it burns or tastes right. In coding, this means the AI writes a tiny bit of code and runs it to see if the computer throws an error.
The Specialized Gadgets (Language-specific Extensions): If the kitchen is specifically Japanese, the apprentice is given a specialized sushi knife. This helps the AI handle the unique, tricky parts of a specific language more efficiently.

The Test: The "Cangjie" Challenge

To prove this works, the researchers used a brand-new, very rare programming language called Cangjie. Because almost no one has ever used it, the AI couldn't "cheat" by using old memories. It had to learn it from scratch using the tools provided.

The Result: Learning by Doing

The results were impressive. The "Apprentice" (ILA-agent) performed much better than the "Expert who just guesses" (Standard AI) or the "Expert who just reads a single page of notes" (RAG).

The researchers even watched the AI's "thought process." They saw that the AI follows a very human pattern:

Phase 1: "I'm lost. Let me search the manual for the basics."
Phase 2: "Okay, I think I found it. Let me try a small test to see if I'm right."
Phase 3: "It failed! Let me look at the manual again to see what I missed, then try one more time."

The Big Picture

This paper moves us away from the idea that AI needs to "know everything" before it starts working. Instead, it shows that if we give AI the right tools to explore and verify, it can become a master of anything—even something that didn't exist yesterday.

Technical Summary: Bridging the Knowledge Void via Inference-time Language Acquisition (ILA)

1. Problem Statement

Large Language Models (LLMs) excel at coding tasks in mainstream programming languages (PLs) due to their extensive pre-training data. However, their proficiency collapses when faced with unfamiliar or nascent programming languages (e.g., newly released or domain-specific languages) where training corpora are scarce.

Existing solutions face significant hurdles:

Finetuning: Requires massive, labor-intensive, and costly data collection, which is often impossible for "cold-start" language ecosystems.
Retrieval-Augmented Generation (RAG): While providing external context, RAG often fails to formulate precise queries for unfamiliar syntax and lacks a verification loop, meaning it cannot validate whether the retrieved knowledge actually works in a runtime environment.

2. Methodology: The ILA-agent Framework

The authors propose a new paradigm: Inference-time Language Acquisition (ILA). Instead of relying on static parametric knowledge, the model is treated as an autonomous agent that learns the language dynamically during the problem-solving process.

The core of the proposed ILA-agent is a suite of behavioral primitives implemented as tools:

Exploration Primitives (Knowledge Acquisition):
- Structural Views (ViewStruct, ViewDetail): Mimic human behavior by allowing the agent to navigate the hierarchical structure of official documentation.
- Global Semantic Search (SemSearch): Uses semantic embeddings rather than keyword matching, allowing the agent to find relevant info even if it doesn't know the exact technical terms of the new language.
Verification Primitives (Knowledge Application):
- Execute: Allows the agent to run arbitrary code snippets to test hypotheses and validate syntax/semantics in a low-stakes environment.
- Submit: Evaluates a complete solution against public test cases to determine functional correctness.
Language-specific Extensions: The framework is extensible via plugins, such as TypeLookup, which provides direct access to type-centric documentation (crucial for statically-typed languages).

The process is modeled as a Partially Observable Markov Decision Process (POMDP), where the agent iteratively moves through states of exploration, application, and verification.

3. Key Contributions

ILA-agent Framework: A general-purpose framework that enables LLMs to incrementally learn and apply unfamiliar programming languages through structured interaction.
Cangjie-bench: A novel, rigorous multi-task benchmark consisting of code generation, translation, and program repair tasks based on Cangjie (a recently released, statically-typed language by Huawei). This provides a "low-resource" testbed where LLMs have minimal prior exposure.
Empirical Analysis: A deep dive into agent trajectories, characterizing how different LLMs (DeepSeek, Qwen, Claude) adopt different "reasoning styles" (e.g., "semantic-first" vs. "structurally grounded").

4. Results and Evaluation

The authors evaluated ILA-agent against specialized finetuning and RAG baselines using frontier models.

Superior Performance: ILA-agent significantly outperformed both task-specific finetuning and RAG baselines across all tasks. For example, on code generation, it showed an absolute accuracy improvement of 16.13% to 20.65% over iterative RAG.
Robustness to Data Scarcity: Even when zero-shot performance was near zero (due to lack of pre-trained knowledge), ILA-agent successfully bridged the gap using only official documentation and an execution environment.
Ablation Insights:
- Removing Verification primitives caused a catastrophic drop in accuracy, proving that exploration without a "verify-refine" loop is insufficient.
- Removing SemSearch led to inefficiency, often causing the agent to exhaust its interaction budget before finding a solution.
Model-Specific Behaviors: While all models followed a general "exploration $\rightarrow$ verification" pattern, they differed in style. For instance, Claude-Sonnet-4.5 tended toward a "semantic-first" approach (more searches), while Qwen3-Max was more "direct" (checking types then immediately executing).

5. Significance

This paper shifts the focus of LLM coding capabilities from knowledge recall (what the model already knows) to knowledge acquisition (how the model can learn on the fly).

The significance lies in its potential to:

Enable Support for Emerging Languages: Developers can use LLMs for brand-new languages immediately upon their release, without waiting for massive datasets to be collected for finetuning.
Provide a Scaffold for Agentic Evaluation: The framework serves as a benchmark for measuring the "agentic" qualities of LLMs—specifically their ability to reason, use tools, and self-correct.
Bridge the Gap in Software Engineering: It moves AI closer to a "human-like" developer who learns by reading documentation and debugging code.