Learning to Learn from Language Feedback with Social Meta-Learning

1. Problem Statement

Large Language Models (LLMs) currently struggle to effectively learn from corrective feedback within a conversational context. Key limitations include:

• Passivity: Models rarely proactively solicit feedback when faced with ambiguity, leading to static, one-sided dialogues.
• Brittleness: Interactions often require perfect initial prompts from users; models fail to adapt when information is revealed incrementally or when the task is underspecified.
• Training Gaps: Current post-training methods focus heavily on single-turn reasoning and static task performance, neglecting the "learning to learn" capability inherent in human social meta-learning (SML).

The authors aim to bridge this gap by developing a methodology that enables LLMs to treat conversation as an in-context learning environment, allowing them to solve problems they cannot resolve in a single turn by actively engaging with a "teacher."

2. Methodology: Social Meta-Learning (SML)

The authors formulate SML as a fine-tuning methodology where static tasks (e.g., math or coding problems) are converted into interactive pedagogical dialogues.

Core Framework

• POMDP Formulation: The interaction is modeled as a Partially Observable Markov Decision Process (POMDP).
  • Agent (Student): The model being trained ($\pi_S$).
  • Teacher: A frozen instance of the same model or a stronger model ($\pi_T$) with privileged information (e.g., ground truth solutions or verifier outputs).
  • State: Composed of public conversation history and the teacher's private knowledge (hidden from the student).
  • Reward: Sparse, binary rewards based on whether the student generates a valid solution by the end of the dialogue.

Training Approaches

The paper explores two primary training strategies:

1. Offline RL (Supervised Fine-Tuning - SFT):
   • Generate a dataset of dialogues.
   • Filter for successful trajectories (where the student solves the problem).
   • Perform SFT on the student's turns from these successful dialogues.
2. Online RL (Group Relative Policy Optimization - GRPO):
   • The model generates a group of conversational trajectories for a given prompt.
   • Trajectories are evaluated, and advantages are calculated based on group-normalized rewards.
   • The policy is updated directly to maximize expected reward, reinforcing successful conversational strategies.

Q-Priming (Two-Stage Training)

To address the specific challenge of ambiguity (where critical information is missing), the authors introduce Q-priming:

• Stage 1 (SFT with Injection): A preliminary SFT stage where the training data is synthetically modified. If a student makes an error, their response is replaced with a generated question.
• Mechanism: The model is given the teacher's private knowledge (ground truth) and prompted to formulate an informative query that would have prevented the error.
• Stage 2 (RL): The model undergoes online RL training. Q-priming ensures the model starts with a strong prior for "exploratory, information-seeking behavior," which is then refined by the RL process.

3. Key Contributions

1. SML Formulation: A novel fine-tuning methodology that frames conversation as an in-context learning environment, proving more effective than single-turn fine-tuning for learning from feedback.
2. Cross-Domain Generalization: Demonstrated that the ability to learn from feedback is a transferable skill. Models trained on math problems using SML show improved performance on coding tasks (and vice versa), even when the feedback mechanism differs (e.g., test case outputs vs. direct answers).
3. Handling Ambiguity: Showed that SML-trained models generalize better to underspecified tasks (where information is revealed incrementally). They make fewer premature answer attempts and are significantly more likely to ask clarifying questions.
4. Q-Priming Strategy: Introduced a two-stage training process that explicitly teaches models to ask questions, significantly enhancing their ability to navigate ambiguity compared to SML alone.

4. Experimental Results

The experiments utilized Gemma-3-12B-IT and Qwen3-8B on Omni-MATH (math) and OpenCodeInstruct/LiveCodeBench (coding).

• In-Domain Performance (RQ1 & RQ2):

  • Online RL > Offline SFT: Online RL yielded substantially greater improvements in learning from feedback compared to offline SFT.
  • Generalization to Length: Models trained on 4-turn dialogues successfully generalized to 10-turn conversations at test time.
  • Teacher Strength: While stronger teachers provide better feedback at inference, a peer-level teacher is sufficient for training the SML skill.

• Cross-Domain Transfer (RQ3):

  • Models trained on Math SML outperformed baselines on Coding tasks (LiveCodeBench) when learning from test-case feedback.
  • Models trained on Code SML showed improved performance on Math tasks.

• Feedback Generalization & Ambiguity (RQ4):

  • On the Lost-in-Conversation benchmark (sharded/underspecified problems), SML models significantly outperformed single-turn baselines.
  • SML models adapted well to incremental information revelation, reducing the burden on users to provide exhaustive initial prompts.

• Behavioral Adaptation (RQ5):

  • Q-Priming Impact: Models with Q-priming were 5x more likely to ask clarifying questions on ambiguous tasks compared to models without it.
  • Premature Attempts: Q-primed models made significantly fewer premature answer attempts, shifting from "presumptive guessing" to "proactive enquiry."

5. Significance

This work presents a scalable path toward developing collaborative AI assistants rather than just instruction-following engines. By mimicking human social meta-learning, the proposed methodology enables LLMs to:

• Self-Correct: Actively seek missing information rather than hallucinating answers.
• Adapt: Handle dynamic, underspecified user intents common in real-world interactions.
• Transfer Skills: Apply reasoning and feedback-learning capabilities across different domains (e.g., math to code).

The findings suggest that the future of robust AI interaction lies not just in larger models, but in training paradigms that explicitly teach models how to learn from others through dialogue.