Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations

Technical Summary: Distilling Examples into Task Instructions for Enhanced In-Context Learning in Real-World B2B Conversations

Problem Statement
While In-Context Learning (ICL) has become the standard for low-resource classification in general Natural Language Processing (NLP), its efficacy in specialized, high-stakes domains like Business-to-Business (B2B) sales remains underexplored. B2B conversations are characterized by semantic complexity, multi-party dynamics, and long sales cycles. The paper identifies a critical limitation in applying standard few-shot ICL to these domains: as the number of few-shot examples increases to improve performance, the concatenation of verbose conversational snippets leads to severe performance degradation. This phenomenon contradicts the conventional assumption that more examples always yield better results. Furthermore, standard ICL lacks transparency; the classification logic is implicit within the examples, making it difficult for human auditors to understand or refine the decision-making process, which is a prerequisite for trust in enterprise applications.

Methodology
The authors propose a novel framework that shifts from example concatenation to automated knowledge extraction. Instead of feeding raw few-shot examples directly into the prompt, the system uses a Large Language Model (LLM) to distill a small set of labeled examples into compact, interpretable representations. This process occurs in two primary stages:

1. Knowledge Extraction: Given a user-defined task intent and a small set of labeled examples, the LLM generates one of two structured outputs:

   • Criteria: Explicit lists of positive and negative conditions that indicate the presence or absence of a concept.
   • Descriptions: Comprehensive natural language narratives defining the classification task, capturing the essence of the concept and its boundaries.
   • Iterative Variants: The framework also explores generating criteria from descriptions (Criteria-De) and descriptions from criteria (Description-Cr) to refine the knowledge representation.

2. Classification: The generated knowledge (criteria or description) replaces the raw few-shot examples in the classification prompt. The target LLM then classifies new conversational snippets based on these explicit instructions rather than implicit pattern matching from examples.

This approach is distinct from existing strategies such as retrieval-based example selection, prompt optimization, or token-level compression (e.g., LLMLingua-2), which often result in incoherent or fragmented outputs. The proposed method maintains coherence while drastically reducing token usage.

The Call Playbook Dataset
To evaluate this approach, the authors introduce the Call Playbook, a new annotated dataset derived from 50 real-world English B2B sales calls (30–90 minutes each). The dataset focuses on five core sales concepts:

• Business Goals
• Decision Criteria
• Decision Makers
• Decision Making Process
• Pain Points

The data is annotated at the snippet level (overlapping sequences of five monologues) with binary labels for each concept. The dataset is balanced, anonymized, and designed to reflect the fluid, multi-stakeholder nature of actual sales negotiations.

Key Results
Experiments conducted across five LLMs (GPT-4o, Claude Sonnet 3.7, Claude Haiku 3, Mistral Large, Mistral Small) and varying few-shot counts (0 to 100 examples) yield the following findings:

• Performance Robustness: Unlike standard few-shot methods, which degrade significantly as example counts increase (dropping from ~71.5% to ~60.7% F1 at 100 shots), the proposed knowledge extraction methods remain robust. Specifically, Description-Ex shows steady improvement, and Criteria-De achieves the largest overall gains (up to 72.2% F1).
• Token Efficiency: The proposed methods achieve a 99% reduction in token usage compared to traditional few-shot learning. While standard methods scale linearly with example count (reaching ~25,000 tokens at 100 shots), the knowledge-based methods remain flat (under 600 tokens for criteria, under 200 for descriptions).
• Superiority over Compression: The method outperforms token-level compression baselines (Selective Context and LLMLingua-2). While compression methods degrade by over 9 F1 points as context grows, the distillation approach maintains or improves performance.
• Cross-Model Distillation: Knowledge generated by larger, more capable models (e.g., Claude Sonnet 3.7) can be effectively transferred to smaller models (e.g., Mistral Small), improving their performance without retraining.
• Human-in-the-Loop: The framework enables direct human refinement. Human annotators were able to modify the generated criteria and descriptions to improve classification performance (up to +2.65% F1), demonstrating the interpretability and utility of the extracted artifacts.

Significance and Claims
The paper claims that its framework addresses the critical needs of transparency, efficiency, and user interaction in real-world NLP applications. By converting opaque example-based learning into explicit, structured task instructions, the method allows for:

1. Auditability: Classification logic is transparent and can be reviewed by humans.
2. Scalability: Significant computational savings allow for faster inference and lower costs, particularly as the volume of data grows.
3. Flexibility: The system supports dynamic refinement of classification logic without retraining models or redesigning prompts from scratch.

The authors position this work as a necessary evolution for B2B analytics, where the complexity of the domain and the need for interpretable AI render traditional few-shot ICL insufficient. The approach offers a pathway to scalable, transparent, and efficient classification in specialized business contexts.