💻 computer science

Predicting Program Comprehension with Foundation Models of Human Cognition

This paper demonstrates that Centaur, a foundation model of human cognition trained on general psychological data, outperforms its base Llama 3.1 model in predicting developer behavior during program comprehension, suggesting that cognitive regularities learned from broad psychological experiments can effectively transfer to complex software engineering tasks.

Original authors: Yannick Lehmen, Marvin Wyrich, Anna-Maria Maurer, Norman Peitek, Marvin Wyrich

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yannick Lehmen, Marvin Wyrich, Anna-Maria Maurer, Norman Peitek, Marvin Wyrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to guess what a friend will say next in a conversation. You could try to memorize every single conversation they've ever had, or you could try to understand the general rules of how humans think, talk, and react. For decades, researchers trying to predict how software developers understand code have been stuck in the first camp: they've built tiny, specific models for specific tasks, or they've tried to measure code "complexity" like counting the number of commas in a sentence. But these methods are like trying to predict a whole movie by only looking at the color of the actors' shoes—they either don't scale up or they just don't match reality very well.

Enter a new idea from the world of psychology. Instead of building a model for just coding, what if we trained a model on how humans act in all kinds of situations? That's exactly what the researchers in this paper did. They took a super-smart AI called Centaur, which had been trained on data from 160 different psychological experiments (like memory games and decision-making puzzles), and asked a big question: Can this "general human brain simulator" understand how developers read code, even though it never saw a line of code during its training?

The Big Discovery
The answer, the paper suggests, is a resounding "yes." When they tested Centaur against 9 different studies involving real developers reading code, Centaur predicted how humans would respond much better than its "parent" model, Llama 3.1, which hadn't been trained on those psychological experiments. In fact, across all the trials, Centaur's predictions were significantly closer to human behavior. Think of it like this: Llama 3.1 is a brilliant student who knows a lot of facts but hasn't learned how people actually think. Centaur is that same student, but after taking a semester of "Human Behavior 101," it suddenly gets how people react, even in a totally new subject like programming.

What It's NOT
Here is the most important part: The paper explicitly rules out a few things.

It's not just about guessing the answer. The researchers were worried that Centaur might just be good at guessing common answers (like "Yes" or "No") because it saw those a lot in its training. But they proved this wrong. When they removed the code and the task instructions from the test, Centaur didn't just rely on "response biases." It actually got better at using the code itself and the task descriptions to figure out the answer.
It's not a magic crystal ball. The paper is careful to say this is a "suggestion" and a "transfer" of patterns, not a final, solved theory of the human mind. They found that for some tricky, confusing code patterns (called "Atoms of Confusion"), Centaur wasn't quite as good as Llama. This suggests that while general human thinking helps, there are still some very specific coding quirks that the model hasn't learned yet.

How They Tested It
To be sure, the researchers played a game of "take away." They fed the models the same code and questions but removed different pieces of information:

The Code: The actual programming snippets.
The Instructions: The text explaining what to do.
The History: What the developer did in previous questions.

They found that Centaur relied less on the history of previous answers (which Llama loved to use) and more on the actual code and instructions. This is a huge clue: it means Centaur learned a deeper way of understanding the task, rather than just following a pattern of "what did I say last time?"

The Numbers
The results were measured using a score called "negative log likelihood," where lower is better. Across all the studies, Centaur achieved an average score of 1.63, while Llama 3.1 scored 1.85. In 7 out of the 9 studies they looked at, Centaur was significantly better. The difference wasn't just a tiny fluke; the statistical "effect size" was 0.17, which the paper notes is a real, measurable improvement.

The Takeaway
This paper doesn't claim to have solved the mystery of how developers think. Instead, it suggests a powerful new path forward. It shows that the way humans understand code isn't a weird, isolated superpower; it's built on the same general cognitive rules that help us play memory games or make decisions in other parts of life. By training AI on general human behavior, we might finally get a tool that can predict how developers will struggle with code, not just by counting lines, but by understanding the human mind behind the keyboard. It's a step toward a future where we can build software tools that are truly "human-aware," grounded in the science of how we actually think.

Technical Summary: Predicting Program Comprehension with Foundation Models of Human Cognition

Problem Statement

Software engineering relies heavily on developers' ability to understand code, yet predicting how they comprehend it remains a significant challenge. Existing approaches suffer from a trade-off between scalability and accuracy:

Controlled experiments offer precise insights but lack scalability and generalizability beyond specific settings.
Proxy measures (e.g., complexity metrics) are scalable but show weak correlations with actual human behavior.
Neurocognitive methods (e.g., fMRI, EEG) provide rich signals but are expensive, difficult to generalize, and limited in scope.

A common limitation across these approaches is the failure to capture transferable regularities in human responses, making it difficult to build predictive models that generalize across different tasks, individuals, and study designs. While psychology has recently shifted toward using "black-box" machine learning models trained on large-scale behavioral data to predict human responses, it remains unclear whether these models can transfer to complex, structured domains like software engineering.

Methodology

The authors evaluate Centaur, a foundation model of human cognition fine-tuned on data from 160 diverse psychological experiments, against its base model, Llama 3.1. The study investigates whether behavioral patterns learned from general cognitive tasks can predict developer behavior in program comprehension.

Dataset Construction

The researchers curated a dataset from 9 previously published program-comprehension studies (out of an initial pool of 95). Selection criteria required:

Accessible replication packages.
Well-structured behavioral data (task descriptions, source code, and participant responses).
Compatibility with natural language prompting (excluding tasks like class diagrams or unconstrained free-text responses).

The final dataset included 9 studies covering topics such as code complexity, indentation, method chains, readability, and "atoms of confusion," involving 208 to 22 participants per study and thousands of trials.

Experimental Design

The evaluation was structured around two Research Questions (RQs):

RQ1: Performance Comparison. The authors compared Centaur and Llama 3.1 on their ability to predict human response distributions. The models were evaluated using Negative Log Likelihood (NLL), measuring how well the models predicted the exact tokens chosen by human participants.
RQ2: Source of Performance (Ablation Studies). To determine if improvements were driven by task-relevant information or generic response biases, the authors conducted ablation studies. They systematically varied the information available in the prompts by removing:
- SEQU: Information about prior trials and participant responses.
- DESC: Task descriptions and instructions.
- CODE: The source code snippets.
This generated 8 distinct prompt categories (e.g., SEQU+CODE+DESC, CODE, NOINFO) to isolate the contribution of each information source.

Key Results

RQ1: Centaur Outperforms Llama 3.1

Overall Alignment: Centaur achieved a significantly lower average NLL (1.63) compared to Llama 3.1 (1.85), indicating a closer alignment with human response patterns ( $p < 0.001$ , Cohen's $d = -0.17$ ).
Study-Level Performance: Centaur significantly outperformed Llama in 7 out of the 9 studies. In the remaining two, one showed no significant difference, and the other showed a negligible decrease in performance for Centaur.
Task Sensitivity: Both models struggled with tasks involving long free-text responses (high NLL), while performing significantly better on tasks with constrained response formats (e.g., Yes/No, Likert scales).

RQ2: Sources of Improvement

Ablation studies revealed that the performance gap between Centaur and Llama is not merely due to learning generic response biases:

Reliance on Prior Responses (SEQU): Llama 3.1 was significantly more reliant on information from prior trials and responses. When SEQU information was removed, Llama's performance degraded more sharply than Centaur's.
Utilization of Task Content (CODE & DESC): Centaur benefited more from the presence of task descriptions and code snippets. Removing these elements caused a larger relative performance drop for Centaur compared to Llama, suggesting Centaur has learned to better utilize task-specific semantic information.
Interaction Effects: In "No Information" (NOINFO) scenarios, Centaur's outputs remained closer to human responses than Llama's, particularly in studies with single-character outputs, suggesting Centaur has learned robust response format regularities independent of input content.

Significance and Claims

The paper claims to provide the first empirical evidence that foundation models of human cognition, trained exclusively on general psychological data (without specific software engineering tasks), can successfully transfer to predict human behavior in program comprehension.

Key implications highlighted by the authors include:

Generalizability of Cognitive Regularities: The results suggest that fundamental cognitive processes (e.g., working memory, attention, semantic processing) learned from simple psychological experiments generalize to complex, structured software engineering tasks.
Shift in Modeling Perspective: The findings support a view of program comprehension not as a set of isolated phenomena, but as an interaction of domain-general cognitive processes. This offers a pathway toward a unified, data-driven framework for modeling developer behavior grounded in cognitive science.
Practical Utility: Such models offer a new, behavior-aware alternative to traditional complexity metrics. They can implicitly incorporate factors often ignored by static analysis (e.g., naming conventions, documentation) and may be used to estimate code comprehension difficulty in real-world workflows, especially as developers increasingly rely on AI coding assistants.
Methodological Complementarity: While not interpretable in the traditional sense, predictive foundation models like Centaur can serve as a complementary tool for hypothesis generation and the refinement of more structured, interpretable theories of program comprehension.

The authors remain modest regarding the scope, noting that their evaluation is limited to specific datasets and response formats, and that the specific cognitive processes driving the transfer require further investigation. They do not claim to have solved program comprehension but rather to have opened a new avenue for studying it through the lens of general human cognition.