💻 computer science

No Resource, No Benchmarks, No Problem? Evaluating and Improving LLMs for Code Generation in No-Resource Languages

This paper addresses the challenge of code generation in no-resource programming languages by introducing new benchmarks and demonstrating that a cost-effective strategy of further pre-training a base model on target language data followed by injecting instruction-following capabilities via weight diff transfer significantly outperforms direct fine-tuning of instruction-tuned models.

Original authors: Alessandro Giagnorio, Alberto Martin-Lopez, Gabriele Bavota

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alessandro Giagnorio, Alberto Martin-Lopez, Gabriele Bavota

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot assistant (a Large Language Model, or LLM) that is an expert programmer. You can ask it to write code in Python or Java, and it's like asking a master chef to cook a classic Italian dish—it knows the recipe by heart because it has read millions of cookbooks (training data) about those specific languages.

But what happens if you ask this robot to cook a dish from a brand-new, obscure cuisine that no one has ever written a cookbook for? Maybe it's a language invented by a company just yesterday, or a specialized tool for a specific industry that hasn't been shared on the internet yet.

This paper is about teaching that robot how to cook these "no-resource" dishes.

The Problem: The Robot is Clueless

The researchers found that when they asked top-tier AI models to write code in these brand-new languages (they tested Gleam and MoonBit), the robots were almost completely lost.

High-Resource Languages (Python/Java): The robot got it right about 80–90% of the time.
Low-Resource Languages (like Lua or R): The robot was okay, getting it right about 50–60% of the time.
No-Resource Languages (Gleam/MoonBit): The robot failed almost 100% of the time.

The Analogy: It's like asking a chef who has only ever cooked with a knife to suddenly use a brand-new, weirdly shaped tool they've never seen. They don't just make a small mistake; they often don't even know how to hold the tool correctly. The errors weren't just "bad cooking"; they were "syntax errors"—the robot couldn't even write the basic structure of the code because it didn't know the grammar of the new language.

The Experiment: How do we teach the robot?

The team tried four different ways to help the robot learn these new languages, treating the new languages like a new subject the robot needs to study for a test.

The "Cheat Sheet" (Few-Shot & RAG): They gave the robot a few examples of code written in the new language or a snippet from the manual right before asking it to write.
- Result: It helped a little bit, like giving a student a few practice problems. The robot got a bit better, but it was still struggling with the basics.
The "Crash Course" (Fine-Tuning): They took the robot and fed it every single piece of code and documentation they could find for the new language, forcing it to relearn everything.
- Result: This worked well. The robot learned the language much better.
The "Deep Dive" (Pre-Training): They took a raw version of the robot (one that hasn't been taught how to follow instructions yet) and fed it all the available data—every code file and every manual page.
- Result: This was the winner. Because they used all the data (not just the neat, organized examples used for fine-tuning), the robot learned the language more deeply. It got the best scores.

The Catch: The "Smart" Robot vs. The "Raw" Robot

Here is the twist. The "Deep Dive" (Pre-Training) worked best, but it had a side effect.

The robot that learned the language deeply was a "raw" model. It knew the language perfectly, but it had forgotten how to listen to human instructions. If you asked it, "Write a function to add two numbers," it might just stare at you or write gibberish because it wasn't trained to follow commands.
The "Smart" robots (Instruction-Tuned models) could follow commands perfectly, but they were terrible at the new language.

The Solution: The "Skill Transfer" (Instruction Transferring)

The researchers came up with a clever hack to get the best of both worlds without spending a fortune on training.

The Metaphor: Imagine you have two twins.

Twin A is a master chef who knows the new cuisine perfectly but is rude and doesn't listen to orders.
Twin B is a polite, obedient waiter who doesn't know the new cuisine at all.

Instead of hiring a new chef or retraining the waiter from scratch, the researchers figured out a way to copy the "politeness" and "listening skills" from Twin B and paste them onto Twin A.

They calculated the "difference" between the polite waiter and the raw chef, and then added that "difference" to the new, language-expert robot.

The Result: They created a robot that is an expert in the new language AND knows how to follow instructions perfectly.
The Bonus: This trick worked so well that a smaller, cheaper robot (8 billion parameters) with this "skill transfer" actually performed better than a much larger, more expensive robot (32 billion parameters) that was just fine-tuned normally.

The Bottom Line

If a company invents a new programming language today, they can't just buy an off-the-shelf AI assistant and expect it to work. The AI will fail.

However, this paper shows a path forward:

Gather whatever tiny bit of code exists for that new language.
Teach a raw AI model that language (Pre-training).
Use a "skill transfer" trick to give that model the ability to listen to instructions.

This allows companies to build their own specialized AI coding assistants for their unique, proprietary languages without needing to spend millions of dollars on massive training runs. They can take a small, smart model and make it a language expert for pennies.

Technical Summary: Evaluating and Improving LLMs for Code Generation in No-Resource Languages

Problem Statement

While Large Language Models (LLMs) have achieved significant success in automating software engineering tasks for high-resource languages (e.g., Python, Java) and, to a lesser extent, low-resource languages (e.g., Lua, R), they face a critical gap when applied to no-resource languages. These are languages that fall outside the pre-training distribution of LLMs, often emerging as proprietary or domain-specific languages in industry (e.g., Gleam, MoonBit). Commercial tools like GitHub Copilot do not support these languages, forcing organizations to develop in-house solutions. However, the lack of training data makes it unclear how to effectively specialize LLMs for these contexts without incurring prohibitive computational costs or losing instruction-following capabilities.

Methodology

Context and Benchmarks

The study evaluates code generation performance across nine languages categorized by resource availability:

High-Resource: Python, Java (>10M GitHub repositories).
Low-Resource: R, Lua, Haskell, Julia, Racket (22k–981k repositories).
No-Resource: Gleam and MoonBit (<3k repositories each, released recently).

To assess these languages, the authors constructed three benchmarks by translating existing datasets (HumanEval, MBPP, and a subset of McEval-Hard) into Gleam and MoonBit. A rigorous translation pipeline was employed, involving automated generation followed by manual inspection to ensure syntactic and semantic equivalence with the original tasks.

Models

Six state-of-the-art LLMs were evaluated, including commercial models (GPT-4o, o3-mini) and open-source models (Qwen 2.5 Coder 32B, Qwen 3 32B/8B), covering both base and instruction-tuned variants.

Experimental Strategies

The study investigates four strategies to improve performance on no-resource languages:

Zero-Shot: Baseline performance without additional context.
In-Context Learning:
- Few-Shot: Providing examples of code in the target language.
- Retrieval-Augmented Generation (RAG): Injecting relevant documentation snippets retrieved from language manuals.
Training-Based Approaches:
- Fine-Tuning: Training instruction-tuned models on collected function-documentation pairs.
- Further Pre-Training: Continuing pre-training on base models using available code and documentation.
Instruction Transferring (Weight Diff): A novel approach where instruction-following capabilities are transferred from an existing instruction-tuned model ( $M_i$ ) to a specialized base model ( $M_{bk}$ ) by computing the weight difference ( $\Delta w = M_i - M_{base}$ ) and adding it to the specialized model. This avoids the high cost of full instruction fine-tuning.

Key Results

RQ1: Impact of Language Popularity

High-Resource: LLMs achieve high pass@1 scores (59–97%).
Low-Resource: Performance remains reasonable (27–87%), often exceeding 50% pass@1, suggesting modern LLMs generalize well even with limited data.
No-Resource: Performance collapses. In a zero-shot setting, pass@1 scores range from 0% to 20%, dropping to near 0% on complex tasks (McEval-Hard).
Error Analysis: Failures in no-resource languages are predominantly syntactic (up to 90% for some models), whereas high-resource failures are mostly semantic. This indicates LLMs lack basic grammar knowledge for these languages.

RQ2: Boosting Performance

In-Context Learning: Few-shot and RAG provide marginal improvements (e.g., increasing pass@1 from ~1% to ~8% on hard tasks) but are insufficient for complex generation. Few-shot outperforms RAG, likely because code examples convey grammar better than documentation.
Fine-Tuning vs. Pre-Training:
- Fine-tuning instruction-tuned models improves performance but is limited by the small amount of high-quality function-documentation pairs available (approx. 0.5M–3.6M tokens).
- Further Pre-Training on base models yields the best results among standard techniques, leveraging the entire corpus of code and documentation (approx. 13M–28M tokens). It significantly outperforms fine-tuning (e.g., 12.47% vs. 3.04% pass@1 on McEval-Hard for Gleam).
- Limitation: Pre-trained base models lose instruction-following capabilities, making them unsuitable as direct coding assistants.

RQ3: Instruction Transferring

Applying the weight diff transfer technique to pre-trained base models successfully restores instruction-following capabilities without full re-training.
This approach yields the highest performance gains, outperforming both fine-tuned instruction models and commercial models with few-shot prompting.
- Example: On Gleam (McEval-Hard), instruction transferring raised pass@1 from 12.47% (pre-trained base) to 26.08%.
- Efficiency: A smaller model (Qwen 3 8B) with instruction transferring outperformed a larger, fine-tuned model (Qwen 3 32B) in several scenarios, demonstrating that strategic knowledge transfer can be more effective than scaling model size.

Significance and Contributions

The paper makes three primary contributions:

Benchmarks: The release of three code generation benchmarks (HumanEval, MBPP, McEval-Hard) translated into two no-resource languages (Gleam and MoonBit), addressing a gap in public evaluation tools for proprietary languages.
Empirical Evidence: A comprehensive study quantifying the performance gap between high-, low-, and no-resource languages, identifying syntax errors as the primary failure mode for no-resource languages.
Practical Solution: The demonstration that further pre-training combined with instruction transferring is a cost-effective, high-performance solution for companies needing to deploy specialized coding assistants for proprietary languages. This approach allows organizations to bypass the computational costs of instruction fine-tuning while achieving superior results compared to standard in-context learning or fine-tuning.

The authors conclude that while no-resource languages remain a significant challenge for out-of-the-box LLMs, the proposed methodology offers a viable path for organizations to develop specialized, in-house AI coding assistants.