Technical Summary: DD-GEPA

Problem Statement

Multi-party chat environments, such as Slack and IRC, frequently contain interleaved dialogues where multiple participants discuss different topics simultaneously. This "entanglement" complicates automatic analysis, as utterances from unrelated threads may be mistaken for relevant context. Dialogue disentanglement aims to partition an entangled sequence of utterances into coherent dialogue clusters based on reply-to relations.

While Large Language Models (LLMs) possess strong contextual reasoning capabilities, their initial application to dialogue disentanglement yielded low accuracy, often falling short of traditional non-LLM methods. Subsequent work demonstrated that prompt design could significantly improve performance, yet the effectiveness of specific prompt components (such as dialogue-level assignment or subsequent context) varies across different LLMs. Furthermore, manually crafting optimal prompts is difficult due to the inherent ambiguity in annotation guidelines and the high sensitivity of LLM performance to specific formatting and instruction details. Consequently, there remains a need for a robust, automatic method to optimize prompts for LLM-based dialogue disentanglement, particularly for open-weight models where privacy concerns necessitate local deployment.

Methodology

The paper proposes DD-GEPA, an automatic prompt optimization framework adapted from GEPA (Agrawal et al., 2026), a method designed for compound AI systems. The core of the approach involves decomposing the dialogue disentanglement prompt into three distinct, optimizable modules:

1. Task Instruction: Defines the task, explains the criteria for assignment decisions, and specifies the evidence to be used.
2. Utterance Representation: Specifies how utterance metadata (index, timestamp, speaker ID) and content are formatted for the LLM (e.g., JSON, labeled text).
3. Output Instruction: Dictates the format of the model's prediction (e.g., specific JSON keys, boolean flags).

Optimization Process:
DD-GEPA maintains a pool of candidate "programs" (combinations of the three modules). The optimization cycle proceeds as follows:

• Selection: A candidate program is selected from the pool using Pareto-based selection, prioritizing those with higher total scores on a validation set while removing dominated candidates.
• Execution Traces: The selected program is run on a minibatch of training instances. Execution traces are collected, including the input, output, reasoning text, and decision correctness. For incorrect decisions, an LLM is prompted to generate a failure explanation.
• Module Update: The optimizer selects one module to update (in round-robin order). Using the collected traces and failure explanations, the LLM generates an improved version of the selected module.
  • For Utterance Representation, the LLM proposes a new format, and an external function generates code to convert raw data into this format.
  • For Output Instruction, the LLM proposes a new output schema, and an external function generates code to parse the model's raw output into the standard evaluation format.
• Evaluation: The new program is evaluated on the minibatch. If it improves performance, it is added to the candidate pool and subsequently evaluated on a larger validation set.

The process iterates for a fixed number of steps, selecting the program with the highest validation score as the final optimized prompt.

Key Contributions

• Automatic Prompt Optimization for Disentanglement: The paper introduces DD-GEPA, the first method to automatically optimize prompts specifically for dialogue disentanglement, addressing the difficulty of manual design for this complex task.
• Modular Decomposition: Unlike previous approaches that treated prompts as monolithic blocks, this work decomposes prompts into task instruction, utterance representation, and output instruction, optimizing them jointly to account for their interdependencies.
• Adaptation of GEPA: The authors adapt the GEPA framework, originally designed for general compound AI systems, to handle the specific constraints of dialogue disentanglement, including the generation of conversion and parsing code for dynamic prompt components.

Experimental Results

Experiments were conducted on the Ubuntu IRC dataset, a standard benchmark for dialogue disentanglement. The study utilized the Qwen3-30B open-weight LLM.

• Performance Gains: The optimized prompt ("Optimum") significantly outperformed both the initial seed prompts and a strong manually designed baseline (reproducing the work of Takada and Mori, 2026) across multiple metrics, including Variation of Information (VI), Adjusted Rand Index (ARI), and F1 scores.
• Comparison with Proprietary Models: While the optimized open-weight model surpassed the manually designed prompt on the same model, it still underperformed compared to proprietary LLMs (e.g., GPT-5.2) using manually crafted prompts.
• Optimization Limits: The optimization trajectory showed rapid improvement in early iterations, with the best program found at the fifth update. Subsequent updates failed to yield further gains, suggesting that the optimization process reached a local plateau or that the training traces lacked sufficient diversity to guide further refinement.

Significance and Claims

The paper claims that DD-GEPA provides a viable pathway to improve the accuracy of LLM-based dialogue disentanglement without requiring manual prompt engineering. By demonstrating that automatic optimization can surpass hand-crafted prompts on the same model architecture, the work highlights the potential of automated methods to unlock the capabilities of open-weight LLMs. This is particularly significant for applications involving chat data with personal information, where open-weight models are preferred over proprietary APIs.

However, the authors maintain a modest stance regarding the ultimate performance ceiling. They acknowledge that even with optimization, the current open-weight approach does not yet match the performance of the strongest non-LLM methods (e.g., DiHRL) or proprietary LLMs. The paper concludes that while DD-GEPA offers a useful basis for future optimization, challenges remain regarding the diversity of training traces and the difficulty of the task itself, as evidenced by the early saturation of optimization gains.