Technical Summary: Contextualized Code Pretraining for Code Generation

1. Problem Statement

Modern code generation models are predominantly trained and evaluated on isolated functions accompanied by natural language descriptions (e.g., HumanEval, MBPP). However, in real-world software development, functions are rarely implemented in isolation. Developers typically write high-level control logic first, which then invokes lower-level utility functions that have not yet been implemented. In these scenarios, the calling context (the code that invokes the target function) provides actionable cues about the expected behavior, data shapes, and error handling requirements that are often absent from natural language descriptions or function signatures.

Existing approaches to repository-level code generation primarily rely on inference-time retrieval (e.g., RepoCoder, GraphCoder) to fetch relevant context. These methods treat context as an external resource to be retrieved and appended to prompts. They face limitations when:

1. No similar code snippets exist in the repository (novel functionality).
2. Retrieval fails to surface the correct context due to indexing or ranking issues.
3. The model is not explicitly trained to interpret and leverage the semantic constraints imposed by the caller.

Consequently, current models often generate code that satisfies a functional description but fails to integrate smoothly with the surrounding usage, leading to implementations that do not adhere to the implicit contracts established by the caller.

2. Methodology

The authors propose Contextualized Code Pretraining, an invocation-aware framework that integrates calling context directly into the pretraining objective of code models. The approach consists of three main components:

A. Invocation-Aware Dataset Construction

The authors developed an automated static-analysis pipeline to extract large-scale caller–callee pairs from real-world Python repositories.

• Extraction: Using Python's ast module, the system parses source files to build a bidirectional call graph. It identifies target functions and extracts the complete bodies of their direct callers as the "invocation-aware context."
• Filtering: To ensure quality, the authors filtered 800 repositories based on popularity, maintenance status, and structural diversity. They excluded algorithmic datasets, test-related code, and incomplete functions.
• Construction: The final corpus contains 235,460 unique target functions. Each training instance pairs a target function with its header, a direct caller's code snippet, and the target's natural language docstring (if available). If a function has multiple callers, each caller-callee pair is treated as a distinct training instance.

B. Pretraining Objective (CallerGen)

The authors introduce CallerGen, a family of models pretrained with an invocation-aware objective. The training task conditions the generation of the target function body on three inputs:

1. The function header (signature).
2. The code snippet of a direct caller (the calling context).
3. The natural language docstring.

The model learns to maximize the likelihood of generating the correct function body given these contextual cues. This approach was applied to two architectures:

• Encoder-Decoder: CodeT5-small (60M) and CodeT5-base (220M).
• Decoder-Only: Qwen2.5-Coder-0.5B.

C. Benchmark: CallerEval

To evaluate models under realistic conditions, the authors constructed CallerEval, a new benchmark featuring 263 function-level generation tasks.

• Design: Tasks are drawn from repositories distinct from the training set to prevent data leakage. Each task includes the target function, its real-world callers, and a behavioral test suite.
• Test Construction: Since many target functions lack existing unit tests, the authors designed a protocol to reconstruct lightweight behavioral tests. These tests are derived from the usage patterns observed in the callers (e.g., how return values are accessed or how exceptions are handled), ensuring the evaluation reflects the "contract" imposed by the caller.
• Evaluation: Performance is measured using Pass@k (functional correctness via execution) and static metrics (CodeBLEU, ROUGE-L).

3. Key Contributions

1. Task Formalization: The paper formalizes "code generation with calling contexts" as a distinct task setting, emphasizing that calling contexts are critical signals for guiding implementation in realistic development scenarios.
2. CallerGen Model: The first code models pretrained with invocation-aware objectives. CallerGen is the first to explicitly condition generation on calling contexts during the pretraining phase, rather than relying solely on inference-time retrieval.
3. CallerEval Benchmark: A new benchmark designed to assess models under caller-driven scenarios, featuring realistic caller-callee pairs and behavior-aware test protocols that go beyond simple functional descriptions.
4. Comprehensive Evaluation: Extensive experiments demonstrating that invocation-aware training improves performance across different model scales and architectures, particularly in scenarios where calling context is available.

4. Experimental Results

The authors evaluated CallerGen against nine state-of-the-art baselines (including CodeGen, PanGu-Coder, DeepSeek-Coder, and Qwen-Coder) on both the public CoderEval benchmark and the new CallerEval.

Performance in Caller-Available Settings

• CallerGen consistently outperformed similarly sized baselines. For instance, CallerGen-base (220M) achieved 16.58% Pass@1 on CallerEval with full inputs (Header + Caller + NL), significantly outperforming CodeGen-350M (8.97%) and PanGu-300M (11.79%).
• CallerGen-qwen (0.5B) achieved 22.81% Pass@1 on CallerEval, surpassing the instruction-tuned Qwen2.5-Coder-0.5B-Instruct (6.16%) by a large margin and narrowing the gap with much larger models (e.g., Qwen2.5-Coder-32B-Instruct at 20.99%).
• The models showed greater robustness; the performance drop when moving from the easier CoderEval to the harder CallerEval was significantly smaller for CallerGen than for baselines.

Performance in Non-Caller Settings

• Even without calling context (Header-only or Header + NL), CallerGen remained competitive, often matching or outperforming similarly sized models. For example, CallerGen-0.5B achieved 13.92% Pass@1 on CallerEval (Header + NL), outperforming Qwen2.5-Coder-0.5B-Instruct (9.13%).
• However, the authors note that all models experienced a performance drop in non-caller settings, highlighting that calling context is a crucial component for high-quality generation.

Ablation and Analysis

• Semantic vs. Length: Ablation studies revealed that the gains come from semantic evidence (how the return value is used, control flow around the call) rather than simply increased prompt length. Replacing caller context with irrelevant code of the same length resulted in lower performance.
• Multi-Caller Supervision: Training with multiple callers per function improved the model's ability to utilize caller evidence at inference time, suggesting that the model learns to reconcile diverse usage constraints.

5. Significance and Claims

The paper claims that calling contexts are a rich and reliable source of semantic intent that are currently underutilized in code generation. By treating calling context as an explicit conditioning signal during pretraining, models can learn to generate code that is not only functionally correct but also behaviorally coherent with its usage environment.

The authors argue that while inference-time retrieval (augmenting prompts with context) is useful, it is insufficient if the model itself has not learned to interpret and leverage invocation constraints. CallerGen demonstrates that invocation-aware pretraining enables models to:

1. Infer implicit behavioral contracts from caller code.
2. Generate implementations that integrate smoothly with existing codebases.
3. Achieve state-of-the-art performance relative to their parameter size, often outperforming significantly larger models.

The work concludes that incorporating calling contexts is essential for advancing code generation toward realistic, repository-level software development scenarios. The authors acknowledge limitations, noting that their current pipeline is specific to Python and relies on static analysis, which may face challenges with dynamic features, but they posit that the core principle of invocation-conditioned generation is language-agnostic.