💻 computer science

Unseen-Codebases-Domain Data Synthesis and Training Based on Code Graphs

This paper proposes UCD-Training, a two-stage framework that leverages code graphs derived from unseen codebases to synthesize reasoning-aware training data, thereby enhancing large language models' ability to understand intrinsic component relationships and reason about correct code compositions in newly released software frameworks.

Original authors: Guangsheng Ou, Qiming Zhang, Sirong Chen, Anji Li, Dong Xu, Tiancheng Luo, Dekun Dai, Cuiyun Gao, Long Wang, Jun Zhou, Mingwei Liu, Zibin Zheng

Published 2026-02-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Guangsheng Ou, Qiming Zhang, Sirong Chen, Anji Li, Dong Xu, Tiancheng Luo, Dekun Dai, Cuiyun Gao, Long Wang, Jun Zhou, Mingwei Liu, Zibin Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years cooking with ingredients from a specific, well-stocked supermarket (like GitHub). You know exactly how to combine tomatoes, basil, and mozzarella to make a perfect pizza because you've seen thousands of recipes using those exact ingredients.

Now, imagine you are suddenly hired by a new restaurant that uses a brand-new, secret ingredient that was just invented yesterday. You've never seen it, you don't know what it tastes like, and you certainly don't know how to combine it with your other ingredients.

If you try to cook using your old skills, you might guess wrong. You might think the new ingredient is a type of cheese when it's actually a spice. You might try to put it in the oven when it needs to be fried. This is exactly what happens when Large Language Models (LLMs) try to write code for new software frameworks they haven't seen before. They "hallucinate" (make things up) because they lack the specific "recipe book" for that new environment.

The paper you shared, "UCD-Training," proposes a clever solution to teach these AI chefs how to cook with these brand-new ingredients, even when they only have the raw ingredient itself to look at, not a recipe.

Here is the breakdown of their method using simple analogies:

The Problem: The "Blank Stare"

When developers use new libraries (like a new version of a database or a new AI framework), the AI doesn't know how to use them.

Old Solution (RAG): This is like giving the chef a search engine while they cook. If they need to know how to use the new spice, they look it up.
- The Flaw: The search engine gives you a list of facts, but it doesn't teach you the intuition of how the spice works with everything else in the kitchen. The chef still feels lost when trying to create a complex dish.
The Goal: We need to actually teach the chef the new rules of the kitchen, not just let them look things up while cooking.

The Solution: UCD-Training (The "Two-Stage Cooking School")

The authors propose a two-step training process that turns raw code into a "smart" understanding of the new framework.

Step 1: Building the "Kitchen Map" (Code Graph)

First, the system takes the raw source code (the new ingredients) and analyzes it to build a map.

The Analogy: Imagine taking apart a new, complex machine and drawing a diagram showing how every gear connects to every other gear.
What it does: It identifies that "File A" depends on "File B," and "Function X" calls "Function Y." It turns a messy pile of code into a structured dependency graph. This map shows the hidden relationships that a human (or AI) might miss just by reading the text.

Step 2: The Two-Stage Training

Once they have the map, they train the AI in two specific phases:

Phase A: "Dependency-Preserving Pre-training" (Learning the Layout)

The Analogy: Before teaching the chef how to cook a meal, you make them walk through the kitchen in a specific order. You say, "First, you must pick up the knife, then the cutting board, then the vegetables." You force them to see the items in the order they are actually used.
The Tech: The AI reads the code files in an order that respects their dependencies (like following the map). This helps the AI memorize the structure of the new framework. It learns, "Oh, I always need to import this library before I can use that function."

Phase B: "Graph-Grounded Fine-Tuning" (Learning the Recipes)
This is the most creative part. Since there are no real "user examples" (recipes) for this brand-new code, the system synthesizes (fakes) them using the map it built. It creates three types of "practice exams":

Single-Hop Relations (The "Vocabulary" Quiz):
- Analogy: "If I have a 'Spoon' and a 'Bowl', how do they connect?"
- The Tech: The AI learns simple connections between two pieces of code. It learns the basic grammar of the new framework.
Compositional API Reasoning (The "Complex Dish" Challenge):
- Analogy: "Here is a list of ingredients. Create a dish that uses a knife, a pan, and a stove together to fry an egg."
- The Tech: This is the big one. The system looks at the map and invents complex scenarios where multiple parts of the code must work together. It forces the AI to figure out how to combine different functions correctly.
- Why it works: Instead of just memorizing a single function, the AI learns the logic of how to build things.
Codebase Utilization (The "Real-World Simulation"):
- Analogy: Taking the test cases the original developers wrote (which are like "quality control checks") and turning them into "Cooking Challenges" for the AI.
- The Tech: It turns existing tests into questions like, "Write a program that passes this specific test." This teaches the AI how the framework is actually used in the real world.

The Secret Sauce: Reasoning Traces
Crucially, when the system generates these practice problems, it doesn't just give the answer. It forces an AI to write out its thought process (the "reasoning trace") explaining why it chose those ingredients.

Analogy: Instead of just showing the finished pizza, the chef explains, "I chose this crust because it holds the sauce better, and I added cheese last because..."
This teaches the AI how to think, not just what to output.

The Results: Why It Matters

The authors tested this on a new benchmark called UnseenCodeBench (a test kitchen with brand-new, secret ingredients).

The Result: The AI trained with UCD-Training was significantly better than AI that just looked things up (RAG) or AI trained on generic data.
The Analogy: The UCD-trained chef could walk into the new restaurant, look at the secret ingredients, and immediately start cooking complex dishes without needing to ask for help. The "search engine" chefs kept getting stuck or making up fake ingredients.
Real-World Impact: Even when the AI had to handle multiple new kitchens at once (multiple languages and frameworks), it still performed better than any other method.

Summary

In short, this paper solves the problem of "AI getting lost in new code" by:

Mapping the new code to understand its structure.
Simulating realistic usage scenarios (recipes) based on that map.
Teaching the AI to reason through those scenarios, turning it from a "searcher" into a "true expert" of the new framework.

It's like upgrading an AI from a tourist with a map and a guidebook to a local who knows the neighborhood by heart.

1. Problem Statement

Large Language Models (LLMs) excel in code generation tasks when trained on mature public code ecosystems (e.g., GitHub). However, they struggle significantly with unseen codebases, such as newly released libraries, proprietary enterprise systems, or domain-specific frameworks.

Challenges:
- Hallucination & Poor Performance: LLMs lack domain-specific knowledge, leading to incorrect component usage, invalid API compositions, and reliance on non-existent functionalities.
- Limitations of Inference-Time Augmentation: Techniques like Retrieval-Augmented Generation (RAG) provide relevant source code snippets but fail to teach the model the intrinsic relationships and latent usage patterns between components. They do not enable the model to reason about correct compositions.
- Data Scarcity for Training: Unlike public domains, unseen codebases often provide only source code implementation. High-quality, usage-oriented examples (demonstrating how components are combined) are scarce or absent, making standard post-training (CPT/SFT) difficult. Existing data synthesis methods (e.g., OSS-Instruct) rely on pre-existing usage patterns, which do not exist for these new codebases.

2. Methodology: UCD-Training

The authors propose UCD-Training, a two-stage training framework that injects domain-specific knowledge into LLMs using only source code from unseen codebases. The core innovation is constructing a Code Graph to drive reasoning-aware data synthesis.

Stage 1: Code Graph Construction

The system parses the source code to build a structured graph where:

Nodes: Files, classes, functions, methods, and global variables.
Edges: Dependencies, calls, includes, and other relations.
This graph captures the structural and semantic relationships within the codebase.

Stage 2: Dependency-Preserving Continued Pre-Training (CPT)

Goal: To help the model memorize the implementation structure and dependencies of the codebase.
Process:
1. Extract file-level dependencies from the code graph to form a Directed Acyclic Graph (DAG).
2. Perform Depth-First Traversal (DFS) starting from nodes with zero in-degree.
3. Concatenate file contents along traversal paths.
4. Apply a sliding-window strategy to ensure files with direct dependencies appear adjacently within the same training context window.
Outcome: A dependency-preserving corpus that teaches the model how code files collaborate, followed by mixing with general code data to prevent catastrophic forgetting.

Stage 3: Graph-Grounded Supervised Fine-Tuning (SFT)

The authors synthesize three types of training data, all augmented with explicit reasoning traces (Chain-of-Thought) derived from the code graph:

Single-Hop Relation Reasoning Data:
- Derived from edges in the code graph (e.g., "Function A calls Function B").
- Synthesized via rule-based mapping and LLM-driven augmentation (including negative samples).
- Includes ground-truth context (full file contents) to teach structural relationships.
Compositional API Reasoning Data:
- The most critical component. Synthesized based on internal test cases of the codebase.
- Generates "examination questions" requiring the combination of multiple APIs to solve functional scenarios.
- Includes strict filtering to ensure semantic coherence and syntactic correctness.
Codebase Utilization Data:
- Transforms internal test cases into function-generation tasks.
- Decomposes test cases into functional implementations and assertions, then generates new tasks requiring the model to replicate the functionality using the correct API invocations.
- Uses the code graph to retrieve all necessary dependencies for the context.

3. Key Contributions

UCD-Training Framework: A novel two-stage pipeline (CPT + SFT) that enables LLMs to adapt to unseen codebases using only source code, without requiring pre-existing usage documentation.
Reasoning-Aware Data Synthesis: Introduces a method to generate high-quality training data (Single-hop, Compositional, Utilization) augmented with explicit reasoning traces grounded in the code graph, addressing the "data scarcity" of unseen domains.
UnseenCodeBench: A new benchmark for evaluating code generation on unseen codebases across C++ and Python. It ensures "unseen" status by selecting projects created after the models' knowledge cutoff dates (post-2025) and includes diverse domains (databases, frameworks, etc.).
Comprehensive Evaluation: Extensive experiments demonstrating superiority over RAG and other training baselines across different languages, model scales (8B–32B), and architectures.

4. Experimental Results

The authors evaluated UCD-Training against baselines including RAG (inference-time), OSS-Instruct, and COTTON (training-time) across multiple models (Qwen family, GPT-5.1, etc.).

Performance Gains:
- UCD-Training consistently outperformed RAG-based baselines. For example, an 8B model trained with UCD-Training achieved a 31.5% Pass@1, surpassing the best RAG baseline (Claude-Sonnet-4.5) by 10.1%.
- Compared to other training-based baselines, UCD-Training showed improvements ranging from 7.2% to 26.1% in Pass@1 on UnseenCodeBench.
- On complex codebases (e.g., sqlgen), UCD-Training significantly reduced hallucinations and improved compilation rates compared to baselines that failed to identify correct dependencies.
Ablation Studies:
- Compositional API Data was identified as the most critical component; removing it caused a massive performance drop (up to 29.7%).
- Dependency-Preserving CPT alone provided a non-trivial boost, proving the value of structural learning.
- Filtering Mechanisms were essential for data quality; removing them led to significant degradation.
Generality & Scalability:
- The method works effectively across C++ and Python.
- Performance scales with model size (8B $\to$ 32B).
- Real-World Scenario: A single model trained on multiple unseen codebases simultaneously achieved a 36.0% Pass@1, outperforming all baselines by 4.7%–23.6%, proving robustness in multi-domain enterprise settings.

5. Significance

This work addresses a critical gap in the deployment of LLMs for enterprise software development, where codebases are often private, new, or highly specialized.

Paradigm Shift: It moves beyond relying on retrieval (RAG) or public data, demonstrating that structural analysis of source code combined with synthetic reasoning data is sufficient to teach models how to use unseen systems.
Practical Impact: The proposed framework allows organizations to rapidly adapt LLMs to their proprietary codebases without needing to manually curate large datasets of usage examples, significantly lowering the barrier for AI-assisted development in closed environments.
Benchmarking: The introduction of UnseenCodeBench provides a standardized, rigorous testbed for future research in unseen domain adaptation.