💻 computer science

Personalizing LLM-Based Conversational Programming Assistants

This paper discusses ongoing and future research aimed at characterizing how cognitive and organizational diversity influences developers' needs, with the goal of exploring personalization strategies to enhance the inclusivity of LLM-based conversational programming assistants.

Original authors: Jonan Richards

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonan Richards

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, all-knowing robot assistant named "Code-Bot." This robot can write computer programs, find bugs, and explain complex code. It's like having a genius engineer sitting next to you 24/7.

But here's the problem: Code-Bot speaks in a very rigid way.

If you ask it a question, it gives you an answer. But sometimes, that answer is perfect for a senior engineer but completely confusing for a beginner. Sometimes, it's too detailed for someone in a rush, or too vague for someone who needs to learn. It's like a teacher who only knows how to teach one specific style of learning, regardless of whether the student is a visual learner, a hands-on learner, or someone who just needs a quick hint.

This paper is about teaching Code-Bot to understand you.

Here is the breakdown of the research in simple terms:

1. The Problem: One Size Does Not Fit All

The author, Jonan, noticed that developers (people who write code) are all different.

The "Cognitive" Difference: Some people think in big pictures; others think in tiny details. Some are experts; others are beginners.
The "Context" Difference: Some people are working alone in a quiet room; others are in a chaotic team meeting with strict company rules.

Currently, Code-Bot treats everyone the same. It's like a restaurant that serves the exact same meal to a hungry athlete, a dieting grandmother, and a picky toddler. It doesn't work well for anyone.

2. The Goal: The "Chameleon" Assistant

The goal of this research is to build a Personalized Programming Assistant. Think of this assistant not as a robot, but as a chameleon.

Just as a chameleon changes its colors to match its environment, this new assistant will change how it talks and helps based on who is using it.

For the Beginner: It might say, "Hey, let's break this down step-by-step. Here's why this line of code works."
For the Expert: It might say, "Here's the optimized function you asked for. No fluff."
For the Stressed Manager: It might say, "I found the bug. Here is the fix. You're good to go."

3. How They Plan to Do It (The Three-Step Recipe)

The author has a three-phase plan to build this chameleon assistant:

Phase 1: The "Get to Know You" Tour

Before the assistant can change, it needs to understand human diversity.

The Experiment: The researcher is watching different types of developers (newbies vs. pros) talk to current AI tools.
The Metaphor: Imagine a detective watching people order coffee. They notice that some people need a long explanation of the bean origin, while others just want "Black coffee, hot, now." The researcher is mapping out these different "ordering styles" to see what makes each person tick.

Phase 2: Designing the "Shape-Shifting" Rules

Now that they know people are different, they need to figure out how the assistant should adapt.

The Challenge: How does the robot know what you need without you telling it?
The Solution: They are testing two methods:
1. The Mind Reader (Implicit): The robot watches what you type and guesses your needs. (e.g., "You asked a simple question, so you probably want a simple answer.")
2. The Dashboard (Explicit): The robot asks, "Hey, do you want me to be a strict teacher or a helpful buddy today?" and lets you set the rules.
The Balance: They want to find the sweet spot where the robot is smart enough to guess, but not so smart that it annoys you by guessing wrong.

Phase 3: Building and Testing the Prototype

Finally, they will build the actual "Chameleon Assistant" and test it.

The Test: They will give the new assistant to a group of developers and see if it makes their lives easier.
The Safety Check: They are very worried about bias. If the robot learns from bad data, it might start treating certain groups of people unfairly (like a teacher who only likes one type of student). They are building special "safety filters" to make sure the robot is fair to everyone, regardless of their background or experience level.

4. Why This Matters

Right now, AI tools are great, but they can be intimidating. If a tool feels like it's designed only for "experts," beginners might feel left out, and companies might lose out on diverse talent.

By making these assistants personalized, the author hopes to:

Make coding more inclusive (so anyone can learn).
Make developers happier and less stressed.
Create tools that feel less like a cold machine and more like a helpful human partner.

In a nutshell: This paper is about taking a "one-size-fits-all" robot and turning it into a "custom-tailored" best friend that knows exactly how to help you specifically, no matter who you are or what you're doing.

1. Problem Statement

Large Language Models (LLMs) have become integral to software engineering (SE) as conversational "programming assistants" (e.g., GitHub Copilot, Tabnine). While they offer technical utility, current tools struggle to effectively serve the diverse needs of all developers.

The Core Issue: There is a significant gap in inclusivity. Challenges in using these assistants are linked to developer experience, learning styles, gender, and cognitive diversity. Furthermore, organizational contexts (team dynamics, policies, lifecycle position) heavily influence how developers interact with these tools.
The Barrier: Effective prompting requires substantial effort and experience. Current "personalized" support via natural language has a high barrier to entry, failing to support cognitive diversity or varying problem-solving styles.
Research Question (RQ): How can an LLM-based programming assistant provide effective personalized support to increase inclusivity regarding cognitive diversity and organizational context?

2. Methodology

The research employs a multi-phase, mixed-method approach combining Human-Computer Interaction (HCI) theory with AI engineering. The work is structured into three main phases:

Theoretical Foundation

The research is grounded in Hassenzahl's UX model (distinguishing between "do-goals" and "be-goals") and Norman's Seven Stages of Action. It posits that personal factors (cognition, experience) and contextual factors (organizational structure) dictate a developer's Needs (psychological state) and Interaction Style (observable behavior), which in turn determine the required support.

Phase I: Understanding Diversity (Empirical Studies)

Study PRE (Preliminary): A within-subject study ( $n=14$ ) testing a code-understanding assistant that used "Theory of Mind" prompting to infer needs. It confirmed that personalization improves perceived clarity but highlighted that LLMs require explicit guidance to personalize effectively.
Study A (Quantitative/Qualitative): A study with $n=27$ diverse participants in a code-change setting using GitHub Copilot Chat. It combines "thinking-aloud" protocols, semi-structured interviews, and chat log analysis to map relationships between cognitive profiles, experience, and interaction styles.
Study B (Case Study): Semi-structured interviews within a single software engineering company to analyze how social and organizational dynamics (e.g., team roles, peer influence) shape needs and interaction styles.

Phase II: Designing Personalization Strategies

Study C (Implicit Personalization): Investigates using NLP and LLM techniques to infer developer needs from interaction styles (chat logs) derived from Study A. Focuses on bias detection and accuracy.
Study D (Explicit Personalization & Human-in-the-Loop): Uses a vignette survey to determine which support aspects should be tailored. It explores mechanisms for users to configure, inspect, and edit inferred needs to balance automation with user control.

Phase III: Implementation and Evaluation

Study E (Evaluation Framework): A position paper proposing a hybrid evaluation method. It advocates for "LLM-as-a-Judge" and automated human-centered evaluation to handle the high frequency of LLM interactions, complementing traditional manual user studies.
Study F (Prototype & Validation): Development of a personalized programming assistant prototype integrating implicit inference and explicit configuration.
- Pilot: $n=10$ diverse participants.
- Final Study: Large-scale user study and automated evaluation to compare the prototype's effectiveness against non-personalized baselines.

3. Key Contributions

The paper outlines three primary contributions:

Explanatory Framework: A conceptual model (Figure 1) detailing how cognitive diversity (experience, problem-solving style) and organizational context interact to shape developer needs and interaction styles.
Personalization Strategies: A design framework for balancing implicit personalization (automatic inference) and explicit personalization (user configuration) to maximize inclusivity while minimizing bias and user effort.
Prototype System: A functional prototype of a personalized programming assistant that adapts its output based on inferred and user-specified needs, alongside a novel evaluation framework combining automated and human-centric metrics.

4. Preliminary Results & Findings

Diversity Exists: Even within a sample of novice developers, interaction styles varied significantly.
Impact of Personalization: In the preliminary study, personalization was perceived to improve the assistant's understanding of queries and the clarity of responses.
Need for Guidance: LLMs cannot automatically personalize effectively without explicit instructions on how to adapt to specific mental states or interaction patterns.
Context Matters: Preliminary data suggests that needs shift based on context (e.g., a junior developer in onboarding prioritizes codebase understanding over raw productivity).

5. Significance and Future Impact

Inclusivity in SE: The research directly addresses the gender and experience gaps in LLM adoption by designing tools that accommodate diverse cognitive styles rather than forcing a "one-size-fits-all" interaction model.
Theoretical Advancement: It bridges the gap between HCI theories of user experience and the practical engineering of LLM-based agents, offering a structured way to handle the "unbounded" nature of conversational programming.
Practical Application: The proposed framework provides actionable insights for:
- Developers: Receiving tailored support that matches their workflow and skill level.
- Tool Providers: Designing more inclusive assistants with built-in personalization mechanisms.
- Organizations: Deploying tools that reduce friction and increase productivity across diverse teams.
Evaluation Innovation: By proposing an automated evaluation framework for conversational agents, the work addresses the scalability challenge of testing LLM tools, which traditional manual studies cannot handle at scale.

Risks and Mitigation

The author acknowledges risks of theoretical drift (mitigated by strong theoretical grounding), overgeneralization (mitigated by reporting sample diversity and avoiding broad assumptions), and bias reinforcement (mitigated by human-in-the-loop feedback and user consent mechanisms).