Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction

This paper evaluates small language models for leader-follower role classification in human-robot interaction, demonstrating that fine-tuned models achieve high accuracy and low latency on edge devices, though performance degrades in one-shot modes due to architectural limitations with increased context.

The Gist

Imagine you are walking through a busy hospital with a helpful robot companion. Sometimes, you know exactly where you're going, and the robot should just follow you like a loyal dog. Other times, you're lost, and the robot needs to take the lead, guiding you to the right room.

The big challenge for engineers is: How does the robot know which role to play right now? Is it time to be the "Leader" or the "Follower"?

For a long time, scientists tried to solve this using "Super-Brains" (Large Language Models). But these are like giant, power-hungry servers that need a super-fast internet connection and a lot of electricity. They are too heavy and slow to carry around on a small robot.

So, this paper asks: Can we use a "Pocket-Sized Brain" (a Small Language Model) instead? And if so, how do we teach it to make the right choice instantly?

Here is the story of their experiment, broken down simply:

1. The Problem: The Robot's Identity Crisis

The researchers wanted a robot that could switch roles instantly based on what you say. But small robots can't carry the "Super-Brains." They need a tiny, efficient model that runs offline (no internet needed) and thinks fast.

2. The Solution: Building a Training Gym

The team realized there was no existing "textbook" for robot conversations about leading and following. So, they built their own.

• The Dataset: They took thousands of normal human conversations and used AI to rewrite them into specific scenarios: "Please lead me to the elevator" (Leader) vs. "I'll walk, you follow" (Follower).
• The "Scarecrow" Test: To test the robot without annoying real humans, they created a "Scarecrow" AI. This fake human would ask confusing questions or give vague answers to see if the robot could still figure out who should be in charge.

3. The Two Teaching Methods

They tried two different ways to teach the tiny robot (Qwen2.5-0.5B):

• Method A: The Cheat Sheet (Prompt Engineering)
  Imagine giving the robot a sticky note that says, "If you hear 'follow me,' you are the follower. If you hear 'lead me,' you are the leader."

  • Result: It was okay, but the robot got confused easily. It was like trying to read a tiny map while running a marathon. It was slow and often made mistakes.

• Method B: The Boot Camp (Fine-Tuning)
  Instead of just giving instructions, they actually trained the robot's brain on thousands of examples until it "learned" the pattern deeply.

  • Result: This was a game-changer. The robot became a pro. It could decide who was leading or following with 86% accuracy and did it incredibly fast (in the blink of an eye—22 milliseconds).

4. The Twist: The "One-Shot" Trap

The researchers thought, "What if we let the robot ask a clarifying question?"

• Zero-Shot: You say "Follow me," and the robot decides immediately. (Success!)
• One-Shot: You say "Follow me," the robot asks, "Are you sure?", you say "Yes," and then it decides.

Here is the catch: When they added that extra step (the back-and-forth conversation), the tiny robot's brain got overwhelmed. It's like asking a person with a very small memory to remember a whole story and a new question and the answer all at once.

• The Result: The accuracy crashed from 86% down to about 51% (basically guessing like a coin flip). The extra conversation created too much "noise" for the tiny brain to handle.

5. The Length Limit

They also tested how long the sentences could be.

• Short sentences: The robot handled them perfectly.
• Long, rambling sentences: The robot started to hallucinate and fail. The tiny brain simply couldn't hold onto the whole story if it got too long.

The Big Takeaway

This paper teaches us three important lessons for the future of robot helpers:

1. Training is better than instructions: For tiny robots, you can't just give them a rulebook (prompts); you have to train them (fine-tuning) to make them fast and accurate.
2. Keep it simple: Tiny robots are great at quick, direct commands. If you make them have a long, complex conversation, they get lost.
3. The Edge is real: We can put smart AI on small, battery-powered robots, but we have to design the conversation carefully. If the robot is too small, it needs short, clear instructions to work its magic.

In short: If you want a robot that can instantly know whether to lead you or follow you, give it a specialized training session, keep the conversation short, and don't expect it to handle a long, complicated chat. It's a "pocket-sized genius" that works best when the job is simple and direct.

Technical Summary

Here is a detailed technical summary of the paper "Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction."

1. Problem Statement

In Human-Robot Interaction (HRI), specifically within assistive and healthcare domains, the leader-follower paradigm is critical for dyadic coordination. The robot must accurately determine whether to lead (provide guidance) or follow (preserve user autonomy) based on natural language inputs.

• The Challenge: While Large Language Models (LLMs) excel at interpreting linguistic ambiguity, their size, latency, and power consumption prevent on-device deployment on resource-constrained mobile robots.
• The Gap: Small Language Models (SLMs) offer a viable alternative for edge computing, but their effectiveness for specific HRI role classification has not been systematically evaluated. Furthermore, there is a lack of datasets specifically designed for leader-follower communication, and the trade-offs between prompt engineering (zero-shot/one-shot) and fine-tuning in this context remain underexplored.

2. Methodology

A. Dataset Construction

To address the lack of domain-specific data, the authors created a novel dataset:

• Source: Derived from the DailyDialog dataset, extracting 415 questions related to guidance and orientation.
• Augmentation: Used three LLMs (DeepSeek, Gemini, GPT-4) to generate synthetic paraphrases, resulting in 5,400 total samples (1,800 per generator).
• Validation: Semantic fidelity was verified using Sentence-BERT (SBERT), showing high cosine similarity (0.78–0.84) between synthetic and original samples.
• Configurations:
  • Zero-Shot: Single user input with a role label.
  • One-Shot: User input + a clarifying question + a simulated (scarecrow) ambiguous response to test multi-turn context handling.
• Split: 315 samples for augmentation training, 100 human-sourced samples reserved for evaluation.

B. Model Selection

• Base Model: Qwen2.5-0.5B (0.5 billion parameters).
• Rationale: Selected for its balance of computational efficiency and instruction-following capabilities, suitable for embedded systems (e.g., NVIDIA Jetson) where latency and memory are critical constraints.

C. Adaptation Strategies

The study compared three approaches:

1. Baseline: Untrained model.
2. Prompt Engineering:
   • Zero-shot: Single system prompt with task instructions.
   • One-shot: Two-step prompting (first to generate a clarifying question, second to classify based on the full context).
3. Fine-Tuning:
   • The model was fine-tuned on the synthetic dataset using the Autotrain framework.
   • A lightweight linear head was added for binary classification (Leader vs. Follower).
   • Training involved 10 epochs, mixed-precision (FP16), and a stratified 20% validation split.
   • For one-shot, two complementary models were trained: one for generating clarifications and one for final classification.

D. Evaluation Protocol

• Metrics: Accuracy, Precision, Recall, F1-score, Tokens/Second (Throughput), and Latency (ms).
• Statistical Rigor: Monte Carlo Cross-Validation (MCCV) with 30 independent iterations to ensure statistical de-biasing.
• Secondary Analysis: Impact of sentence length (character count) on classification accuracy.

3. Key Results

A. Performance Comparison (Zero-Shot vs. One-Shot)

Strategy	Zero-Shot Accuracy	One-Shot Accuracy	Latency (ms)
Baseline	55.00%	48.00%	~33–41
Prompt Eng.	53.87%	45.07%	~111–213
Fine-Tuning	86.66%	51.65%	~22

• Zero-Shot Success: Fine-tuning significantly outperformed both baseline and prompt engineering (86.66% vs. ~54%, p < 0.001). It achieved robust precision and recall, whereas baseline/prompt methods were overly conservative (high precision, extremely low recall).
• One-Shot Failure: Performance degraded drastically for all methods in one-shot mode. Fine-tuning dropped to 51.65% (near chance level), with recall collapsing to 8.90%. This suggests the 0.5B model cannot handle the increased context length and semantic complexity of multi-turn dialogue.
• Efficiency: Fine-tuned models were the most efficient, achieving the lowest latency (~22 ms) and highest throughput. Prompt engineering incurred significantly higher latency, especially in one-shot mode due to context window management.

B. Sentence Length Analysis

• Zero-Shot: Accuracy remained robust (>80%) across varying sentence lengths, indicating the model captures intent well in single-turn interactions.
• One-Shot: Accuracy showed a progressive decline, dropping from ~65% for short inputs to ~25% for long inputs. This confirms that verbose, multi-turn dialogues overwhelm the parameter capacity of sub-1B models.

4. Key Contributions

1. Novel Dataset: The first publicly available dataset specifically tailored for leader-follower HRI communication, enabling reproducible benchmarking.
2. Adaptation Benchmark: A systematic comparison showing that fine-tuning is superior to prompt engineering for SLMs in HRI role classification, offering both higher accuracy and lower latency.
3. Architectural Insight: Identification of a critical trade-off: while one-shot interactions mimic natural human dialogue, they exceed the capacity of extremely small models (0.5B), leading to coordination breakdowns.

5. Significance and Implications

• Edge Deployment Viability: The study confirms that fine-tuned SLMs (like Qwen2.5-0.5B) are a viable, low-latency solution for real-time role assignment in assistive robotics, provided the interaction remains concise (zero-shot).
• Design Guidelines: For edge-constrained robots, concise interaction design is crucial. Relying on multi-turn clarification (one-shot) with sub-1B models introduces noise that degrades reliability.
• Future Directions: To enable natural multi-turn HRI on the edge, future work requires:
  • Specialized context-pruning pipelines.
  • Larger or multimodal datasets with human annotations.
  • Evaluation of "scarecrow" simulation fidelity and human-in-the-loop studies to validate complex dialogue reliability.

Conclusion: The paper demonstrates that while fine-tuned SLMs are highly effective for direct, single-turn role classification on edge devices, current small architectures struggle with the semantic complexity of multi-turn dialogues, necessitating a shift in interaction design or model architecture for advanced HRI scenarios.