💬 NLP

Learning to Commit: Generating Organic Pull Requests via Online Repository Memory

The paper introduces "Learning to Commit," a framework that enhances the organicity of LLM-generated pull requests by training agents on historical repository commits via an Online Repository Memory system, enabling them to learn project-specific conventions and architectural patterns to produce code that aligns with real-world maintainer expectations.

Original authors: Mo Li, L. H. Xu, Qitai Tan, Ting Cao, Yunxin Liu

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mo Li, L. H. Xu, Qitai Tan, Ting Cao, Yunxin Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Alien" Coder

Imagine you hire a brilliant new software engineer who has read every coding book in the world. They are a genius at logic and can fix almost any bug instantly.

However, when they submit their first piece of work, the team lead rejects it. Why?

They re-invented a wheel that the team already built.
They named variables in a way that confuses everyone else.
They broke a hidden rule about how the building is structured.

The code works perfectly, but it feels alien. It looks like it was written by someone who has never actually lived in that specific office. Current AI coding agents are like this: they are smart, but they don't know the "culture" or the "history" of the specific project they are working on. They just guess based on general knowledge.

The Solution: "Learning to Commit"

The authors of this paper created a new framework called Learning to Commit. Think of it as a strict, immersive onboarding program for an AI before it is allowed to write a single line of code for a new project.

Instead of just handing the AI the current code and saying, "Fix this bug," the AI is forced to go back in time and learn from the project's history.

How It Works: The "Time-Travel Intern" Analogy

Imagine the AI is a new intern at a famous architecture firm.

1. The Setup (The Time Split)
The firm has a strict rule: The intern cannot see the blueprints for the future buildings. They can only study the past 50 years of completed projects.

2. Step 1: The "Blind" Practice (Contrastive Reflection)
The intern is given a description of a problem from 10 years ago (e.g., "The elevator was too slow").

The Attempt: The intern tries to solve it using only their general brain power. They write a solution.
The Reveal: The boss reveals the actual solution the senior architects used back then.
The Lesson: The intern compares their solution to the real one.
- Oh, I used a heavy metal gear, but they used a lightweight spring.
- Oh, I put the motor in the basement, but they put it on the roof because of the wind.
- Oh, I named the file elevator_fix.py, but they always use lift_system_v2.py.

The intern writes these lessons down in a Personal Skill Notebook. They don't just memorize the answer; they memorize the style, the hidden rules, and the preferred tools of this specific firm.

3. Step 2: The Real Test (Skill-Conditioned Resolution)
Now, a brand new problem arrives (a future task the intern has never seen).

The intern opens their Skill Notebook.
They remember: "In this firm, we never put motors in the basement," and "We always use springs for this."
They solve the new problem, but this time, their solution looks like it was written by a senior architect who has been there for 20 years. It fits perfectly.

Why This is a Big Deal

Previous AI tests were like giving a student a math test and checking if they got the right number. This paper adds a new layer: Did they solve it the way our team does?

The researchers tested this on a real, complex software project. They found that when the AI used this "Time-Travel Intern" method:

It found the right files faster: It didn't waste time looking in the wrong rooms.
It wrote less "bloat": It didn't write 50 lines of code when 10 would do, because it knew the team's shortcuts.
It felt "organic": The code looked like it grew naturally from the project, rather than being pasted in from the outside.

The Takeaway

The paper argues that to make AI truly useful for real-world software, we can't just make it smarter at coding. We have to make it better at learning the specific culture of a project.

By forcing the AI to "fail" on past problems and learn from the experts' corrections, it builds a memory of the project's unique personality. This turns a generic, robotic coder into a seasoned team member who fits right in.

1. Problem Statement

While Large Language Model (LLM) based coding agents perform well on controlled benchmarks (e.g., SWE-bench), they frequently fail in real-world software engineering scenarios. The primary issue is not functional incorrectness but a lack of "organicity."

The "Alien" Code Problem: AI-generated code often ignores project-specific conventions, duplicates existing internal APIs, and violates implicit architectural constraints.
Limitations of Current Evaluation: Existing benchmarks treat tasks as isolated, stateless snapshots, ignoring the chronological evolution of a repository. They fail to account for the "onboarding" process where human developers learn a codebase's history, style, and hidden patterns before contributing.
Data Leakage & Memory Gaps: Current approaches often suffer from pre-training data leakage or rely on unsupervised self-reflection, which leads to error accumulation ("garbage-in-garbage-out") rather than genuine learning of repository-specific norms.

2. Methodology: Learning to Commit

The authors propose Learning to Commit, a framework centered on Online Repository Memory. The core philosophy is that "organicity" is learnable from commit history. The framework operates in two distinct phases using a strict chronological split of a repository's commit history ( $C^-$ for learning, $C^+$ for evaluation).

Phase 1: Repository Onboarding (Learning Phase)

The agent undergoes a supervised contrastive reflection process on historical commits ( $C^-$ ):

Blind Attempt: The agent receives a repository snapshot and a synthetic issue description (derived from the commit message but stripped of implementation details). It attempts to solve the task using standard tools (reading, searching, editing) without prior knowledge of the solution.
Oracle Revelation: The ground-truth "Oracle Diff" (the actual accepted commit by human experts) is revealed.
Contrastive Reflection: The agent compares its blind attempt ( $\hat{\Delta}_t$ ) against the Oracle Diff ( $\Delta_t$ ). It identifies discrepancies in file localization, logic, API usage, and style.
Skill Extraction & Update: The "gap" between the attempt and the oracle serves as dense supervision. The agent updates a structured Skill Document ( $M$ ) via CRUD operations:
- Create: New patterns for unknown conventions.
- Revise: Corrections for partially correct logic.
- Deprecate: Removing patterns contradicted by expert evidence.
- Content: $M$ captures coding style, internal API usage, architectural boundaries, and maintainer preferences.

Phase 2: Skill-Conditioned Resolution (Solve Phase)

When a new, unseen future task ( $d_k \in C^+$ ) arrives:

The agent receives the current repository snapshot, the task description, and the accumulated Skill Document ( $M$ ).
It autonomously resolves the task, conditioning its tool use and code generation on the patterns stored in $M$ .
The goal is to produce a patch that aligns with the repository's evolution rather than generic pre-training priors.

3. Key Contributions

Formalization of Repository-Personalized Adaptation: The paper establishes a rigorous paradigm for agents to learn and adapt to evolving codebases over time, moving beyond static context windows.
Training-Free Online Skill Extraction: A novel method using Supervised Contrastive Reflection. Instead of fine-tuning weights, the agent distills abstract, reusable skills by comparing its own failures against expert oracle diffs, creating a continuously updatable memory.
Strict Time-Split Benchmark: The authors construct a benchmark on an expert-maintained internal repository with a strict temporal split to guarantee zero data leakage. Evaluation tasks are drawn from genuinely future commits that could not have influenced the skill-building phase.
Multi-Dimensional Evaluation: Moving beyond simple test-pass rates, the evaluation includes:
- Deterministic Metrics: File IoU (localization), Trajectory Steps (efficiency), and Line Deviation Ratio (patch bloat).
- LLM Judge Metrics: Scope alignment, logic similarity, redundancy/hallucination, and code style consistency.

4. Experimental Results

The framework was evaluated on an internal reinforcement learning training repository (386 high-quality commits, split into 24 learning and 7 test tasks).

Localization Accuracy (File IoU): The skill-conditioned agent significantly outperformed the baseline. In the best setting (seq-all), File IoU improved from 61% (baseline) to 80% (skill agent).
Efficiency: The skill agent required fewer tool calls (e.g., 56.8 steps vs. 71.9 in seq-all), indicating better navigation of the codebase.
Patch Bloat: The skill agent produced patches closer in size to the oracle (lower Line Deviation Ratio), demonstrating better reuse of internal APIs and less redundant code.
LLM Judge Win Rates: In pairwise comparisons, the skill-conditioned agent won 54–58% of the time against the baseline.
- Strongest Gains: Logic similarity (50% win rate) and Redundancy reduction (54% win rate).
- Weakest Gains: Fine-grained scope alignment and code style showed marginal improvements, suggesting surface-level stylistic nuances may require further refinement.
Learning Modes: Sequential learning with full-corpus assignment (seq-all) yielded the most coherent knowledge, though targeted category-specific learning (par-bycat) also achieved high localization accuracy.

5. Significance and Future Work

Bridging the Gap: This work addresses the critical gap between "functionally correct" code and "maintainable, organic" code. It demonstrates that agents can learn to behave like seasoned contributors by internalizing repository history.
New Evaluation Standard: It introduces a rigorous, time-split evaluation protocol that prevents data leakage and prioritizes "organicity" as a first-class objective, challenging the current reliance on isolated test-pass metrics.
Future Directions:
- Scaling the pipeline to diverse, high-quality public GitHub repositories.
- Developing more robust, non-LLM-based metrics for evaluating human coding preferences.
- Investigating the transferability of repository-specific skills to broader, general coding tasks.

In conclusion, Learning to Commit proves that equipping coding agents with an Online Repository Memory derived from supervised contrastive reflection allows them to generate pull requests that are not only correct but also culturally and architecturally aligned with the target project.