Understanding Conversational Patterns in Multi-agent Programming: A Case Study on Fibonacci Game Development

Technical Summary: Understanding Conversational Patterns in Multi-agent Programming

Problem Statement
While Large Language Models (LLMs) are increasingly applied to software engineering (SE), their potential for autonomous, role-oriented collaboration remains underexplored. Existing evaluations often focus on short, self-contained tasks or single-agent performance, offering limited insight into how multiple agents coordinate over sustained interactions. Empirical evidence suggests that unstructured multi-agent interactions can lead to error propagation, premature consensus on incorrect solutions, or prolonged disagreement that prevents convergence. There is a critical need to understand the conversational dynamics—specifically how agents maintain role alignment and converge on solutions—to build reliable autonomous SE systems.

Methodology
The authors conducted an exploratory empirical study using a controlled, lightweight agentic framework to analyze interactions between two role-specialized agents: a Designer (providing design critique and suggestions) and a Programmer (generating C code).

• Task: The agents were tasked with writing a mathematical game of Fibonacci in C. This task was selected for its well-scoped nature, allowing for design-level improvements without overwhelming complexity.
• Models: The study utilized 12 distinct agent pairs formed from 7 open-source LLMs (Gemma 2, Gemma 3, LLaMA 3.2, LLaMA 3.3, DeepSeek-R1, MiniCPM, and Qwen3). The combinations included symmetric pairs (same model), asymmetric pairs (different sizes), and cross-architecture pairs.
• Framework: A Python-based controller managed deterministic turn-taking, enforced termination conditions (capped at 10,000 iterations), and logged all interactions. Temperature was set to 0.5 to balance variability and stability.
• Analysis: The study employed a mixed-method approach:
  • Qualitative: Thematic analysis of conversation logs to identify Success/Failure Patterns (SFP) and Conversational Behavior Patterns (CBP), such as echoing, shallow interaction, and cross-lingual drift.
  • Quantitative: Measurement of conversation length, semantic alignment using BLEU and ROUGE scores, and compilation success rates using GCC.

Key Contributions
The paper makes three primary contributions to the understanding of multi-agent programming:

1. Efficiency: Analysis of solution convergence, quantifying how quickly different agent pairs progress toward a correct solution and identifying the conversational patterns associated with this journey.
2. Consistency: Measurement of semantic alignment between Designer and Programmer roles using BLEU and ROUGE scores to determine how reliably agents remain on-task and coherent across turns.
3. Effectiveness: Evaluation of the frequency with which agent interactions lead to a correct and compilable C program.

Results and Analysis
The study revealed significant variability in multi-agent performance based on model pairing:

• Convergence Patterns: The DeepSeek-R1:DeepSeek-R1 pair was unique in converging to the correct solution from the first iteration and sustaining it. Conversely, pairs like LLaMA 3.2:LLaMA 3.2 and Qwen3:Qwen3 demonstrated strong role alignment but diverged from the correct solution. Other pairs failed to initiate or converge on the task entirely.
• Conversational Behaviors:
  • Echoing: Many pairs exhibited "echoing" (repetitive responses). Notably, the DeepSeek-R1:DeepSeek-R1 pair used repetition as a mechanism for incremental refinement rather than failure.
  • Divergence: Several pairs that started correctly eventually diverged after a specific number of iterations (e.g., 3, 21, or 69 turns), suggesting that once divergence begins, recovery is unlikely.
  • Topic Drift: While Fibonacci-related discussions dominated, many pairs drifted into unrelated topics (sorting, header files, or generic conversation) or different programming languages (C++, Python).
• Compilation Success: LLaMA 3.2:LLaMA 3.2 and Qwen3:Qwen3 achieved 100% compilation success for generated C code. In contrast, pairs involving DeepSeek-R1 often resulted in "No Code Found" cases (due to the agents providing instructions rather than code) or lower compilation success rates, despite DeepSeek-R1:DeepSeek-R1 being the only pair to find the correct logical solution.
• Role Alignment: High semantic alignment (high BLEU/ROUGE) did not guarantee correctness. Some pairs with strong alignment failed to solve the problem, while the successful pair (DeepSeek-R1:DeepSeek-R1) exhibited semantic echoing and inconsistent role alignment in terms of traditional metrics.

Significance and Claims
The authors position this work as an initial empirical foundation for improving multi-agent LLM collaboration in programming contexts. The study claims that:

• Interaction Dynamics Matter: Multi-agent performance depends more on role separation, pair compatibility, and interaction dynamics than on model size or lexical similarity alone.
• Early Signals Predict Outcomes: The emergence of a correct solution early in the interaction is a strong predictor of eventual success, whereas divergence rarely recovers once initiated.
• Need for Monitoring: Simply allowing agents to interact is insufficient. The findings highlight the necessity for explicit stop conditions, monitoring mechanisms, and calibrated termination strategies in scalable autonomous SE systems.
• Practical Utility: By identifying conversational patterns like repetition, divergence, or stalled convergence, the study offers insights for SE tool developers to trigger early fallback strategies (e.g., model switching or human intervention) and assists organizations in selecting open-source models for internal AI programming assistants.

The paper concludes that while current tools support developers in specific tasks, future Autonomous SE systems require agents that can communicate and coordinate roles without direct human intervention, a capability that requires further research into convergence and stop conditions.