Technical Summary: Synthetic Function Demonstrations Improve Generation in Low-Resource Programming Languages

Problem Statement

Large Language Models (LLMs) trained on code have achieved high accuracy in High-Resource Programming Languages (HRPLs) like Python, largely due to abundant training data containing human-annotated explanatory comments. However, performance degrades significantly in Low-Resource Programming Languages (LRPLs), such as Excel Formulas, Welsh, or niche Domain-Specific Languages (DSLs). The primary bottleneck is the scarcity of naturally occurring training data that pairs code with natural language explanations.

Existing methods for augmenting data, such as using State-of-the-Art (SoTA) LLMs to generate synthetic code, struggle in LRPLs because even powerful models (e.g., GPT-4o) lack sufficient domain expertise to generate high-quality code in these languages without prior training. Furthermore, standard Retrieval-Augmented Generation (RAG) approaches often yield only modest improvements in unfamiliar domains because the models lack the foundational understanding to effectively utilize retrieved information.

Methodology

The authors propose a novel pipeline to generate "textbook-quality" synthetic training data for LRPLs. This approach does not require the teacher model to be an expert in the target domain but instead leverages the model's general programming reasoning capabilities combined with specific domain documentation. The pipeline consists of four steps:

1. Prepare the Function Library: The authors collect documentation for the target LRPL's standard library. For their case study (Excel), they identified the 100 most frequently used functions based on a sample of public spreadsheets and gathered their official Microsoft support documentation, which includes argument vectors, types, and toy examples.
2. Sample Relevant Data Contexts: To ground the synthetic generation in realistic scenarios, the authors sample unannotated data tables (e.g., from Wikipedia's WikiTableQuestions dataset). They use a teacher model (GPT-4o) to select the most appropriate table for each function, ensuring the function can be logically executed on that specific data structure (e.g., matching numeric functions to tables with numeric data).
3. Generate Synthetic Problems: The teacher model is prompted to generate a series of synthetic demonstrations for each function. Crucially, the generation process is systematic: for each function, the model generates a set of examples where each example demonstrates the usage of a specific argument slot (including optional arguments). Each example includes:
   • A natural language question (query).
   • A step-by-step explanation of the logic.
   • The executable Excel formula.
   • The expected answer.
     The output is structured in JSON to facilitate automated compilation into a training curriculum.
4. Validate Generations: Since the synthetic data is AI-generated, it may contain errors. The authors employ a two-step validation process:
   • Execution Check: Filter out any samples where the generated Excel formula fails to execute.
   • Cross-Language Verification: Generate parallel solutions for the same questions using Python (an HRPL where the model is more reliable). The authors execute both the Excel and Python solutions and retain samples where the results match. This process retains approximately 50% of the synthetic generations, significantly improving data quality.

The resulting dataset is used to fine-tune "student" models (Qwen 2.5 and Llama 2 families) on the task of answering questions using Excel formulas.

Key Contributions

1. A Protocol for Synthetic Data Synthesis: The paper defines a general protocol for generating domain-targeted training data for LRPLs. It demonstrates that a teacher model with strong instruction-following capabilities, augmented by function documentation and sampled data contexts, can produce high-quality synthetic examples without needing deep prior expertise in the target domain.
2. Superiority of Fine-Tuning over RAG: The study provides empirical evidence that fine-tuning on synthetic function demonstrations significantly outperforms RAG-based approaches. While RAG (even with oracle-retrieved function signatures) offers inconsistent and marginal gains, fine-tuning allows models to internalize the syntax and semantics of the LRPL.
3. Transfer of General Programming Skills: The results indicate that while pre-training on HRPLs does not directly enable Excel generation, it provides a foundation of general problem-solving skills. Fine-tuning on the synthetic LRPL data effectively transfers these general skills to the specific domain, improving performance by over 10% on table-based QA tasks.

Results

The authors evaluated their approach on two datasets: WikiTableQuestions (WikiTQ) and TAT-QA (a financial domain dataset).

• Performance Gains: Fine-tuning on the synthetic data (FTSyn-QA) resulted in significant improvements across most student models. For instance, on WikiTQ, the Qwen2.5-coder 3B model improved from a baseline of 15.34% to 28.64% Execution Match (EM). The 14B variant improved from 46.95% to 54.93%.
• Comparison with Baselines: The fine-tuned models outperformed both the base models and models augmented with RAG (both naive and oracle-retrieved). Notably, the fine-tuned models surpassed the performance of GPT-4o on the WikiTQ subset (58.2% baseline vs. 54.93% for the 14B fine-tuned model, with the 3B fine-tuned model reaching 28.64% vs. 15.34% baseline).
• Error Analysis: Qualitative analysis revealed that fine-tuning primarily reduced errors related to "Table Indexing" (reading the table correctly) and "Formula Syntax." The models learned to construct valid formulas for individual functions. However, the synthetic data, which focused on single-function demonstrations, led to a slight increase in "Plan Logic" errors, as models tended to generate single-function solutions even when multi-step reasoning might be required.
• Model Scale and Specialization: Larger models generally performed better. Code-pretrained models (e.g., Qwen2.5-Coder) showed greater gains from fine-tuning compared to non-specialized counterparts, suggesting that general coding skills learned in HRPLs transfer well to LRPLs when supported by targeted fine-tuning.

Significance and Claims

The paper claims that generating textbook-quality synthetic examples is a viable and effective strategy for adapting LLMs to low-resource programming languages. The significance lies in the method's simplicity and automation: it requires only public documentation, a general-purpose teacher model, and unannotated data contexts, avoiding the need for expensive human annotation or a teacher model that is already an expert in the target LRPL.

The authors posit that this approach addresses the "long-tail" problem of LRPLs, where natural data is insufficient for training. By leveraging the general reasoning capabilities of pre-trained models and grounding them in specific function semantics through synthetic data, the method enables rapid adaptation to new domains. The paper concludes that while the current method focuses on individual function demonstrations (which may bias models toward simpler solutions), it establishes a robust foundation for future work in synthetic data generation for LLM pretraining and adaptation.