Agile Deliberation: Concept Deliberation for Subjective Visual Classification

1. Problem Statement

Traditional computer vision pipelines rely on objective, well-defined ground truths (e.g., "dog," "car"). However, many real-world applications, such as content moderation and personalized curation, require the classification of subjective concepts (e.g., "unsafe imagery," "gourmet food," "healthy food").

• The Core Challenge: Users often possess vague, evolving mental models of these subjective concepts. They cannot provide a static, perfect definition upfront.
• Limitations of Existing Methods:
  • Crowdsourcing/Fixed Taxonomies: Assume a single ground truth, failing to capture conceptual variability.
  • Agile Modeling/Manual Labeling: Assume the user starts with a clear concept definition and iteratively labels data. They do not support the process of refining the concept itself.
  • LLM/VLM Prompting: Domain experts can write prompts for Vision-Language Models (VLMs), but without a structured deliberation process, these prompts often fail to capture nuanced decision boundaries, leading to arbitrary ambiguity resolution by the model.
  • Standard Active Learning: Typically selects samples based on model uncertainty (probability near 0.5). This is ineffective for VLMs because their confidence scores are often uncalibrated with human understanding, and they may miss semantically ambiguous "borderline" cases that humans find difficult.

2. Methodology: Agile Deliberation Framework

The authors propose Agile Deliberation, a human-in-the-loop framework designed to support the evolution of subjective concept definitions. The system operates through two distinct stages, integrating user feedback directly into the training loop of a VLM-based classifier.

A. System Architecture

The framework takes a subjective concept name ($s$) and an unlabeled dataset ($D$) as input. It outputs a structured concept definition ($d$) and a classifier ($f_d$) that maps images to a probability of being "in-scope."

The process is formalized as an iterative optimization where the definition $d_{t+1}$ is chosen to maximize the F1-score on the accumulated labeled set $\mathcal{L}_t$:
$$d_{t+1} = \arg \max_{d' \in \mathcal{C}_t} F_1(f_{d'}, \mathcal{L}_t)$$

B. Stage 1: Concept Scoping

This stage decomposes a user's initial vague concept into a structured hierarchy.

1. Decomposition: Using prompt-chained reasoning (LLMs), the system breaks the composite concept into unit concepts (e.g., "healthy food" $\to$ "healthy dish" + "healthy beverage").
2. Sub-concept Expansion: For each unit, the system generates candidate positive and negative sub-concepts (e.g., "fresh fruit" vs. "fried fast food").
3. User Validation: The system retrieves representative images for these candidates. The user selects which sub-concepts to include, forming an initial structured definition ($d_0$) and a seed classifier.

C. Stage 2: Concept Iteration

This stage refines the definition by exposing the user to semantically borderline examples rather than just model-uncertain ones.

1. Borderline Image Retrieval:
   • Instead of standard uncertainty sampling, the system generates queries for "ambiguous" categories (e.g., "salad with heavy mayo").
   • It retrieves images, de-duplicates them, and clusters them based on visual features.
   • Multi-Armed Bandit Selection: The system uses an Upper Confidence Bound (UCB) algorithm to select which cluster of borderline images to show next. It prioritizes clusters where the model disagrees with the user or where user feedback has historically been most informative.
   • Ambiguity Mining: Within a selected cluster, the system ensures the batch focuses on a single coherent ambiguity dimension (e.g., portion size) to facilitate focused reflection.
2. User Labeling & Feedback:
   • Users label images as in-scope/out-of-scope and provide textual justifications for disagreements.
   • The interface juxtaposes the VLM's prediction and rationale with the user's label.
3. Concept Refinement (Prompt Optimization):
   • User feedback is expanded into explicit rationales.
   • An LLM synthesizes multiple candidate definitions ($\mathcal{C}_t$) that incorporate these new constraints (e.g., "exclude dishes with >50% cream").
   • The system evaluates candidates against the full labeled history and greedily selects the best performing definition ($d_{t+1}$) for the next round.

3. Key Contributions

1. Operationalizing Concept Deliberation: The paper identifies and formalizes the "concept deliberation" workflow used by human experts (scoping $\to$ iterating on edge cases) into a scalable software framework.
2. Semantic Borderline Retrieval: A novel retrieval strategy that targets semantic ambiguity (via LLM-generated queries and clustering) rather than probabilistic uncertainty, addressing the calibration gap between VLMs and human intent.
3. Human-in-the-Loop Prompt Optimization: A method that treats the concept definition as a dynamic prompt, continuously optimizing it based on structured user feedback rather than static few-shot examples.
4. Evaluation Protocol: The authors move beyond static benchmarks, conducting 18 live user sessions (1.5 hours each) to measure the system's ability to support the dynamic evolution of user intent.

4. Experimental Results

The study involved 9 non-expert participants (18 sessions total) testing two concepts: "Paid to Play" (content moderation) and "Healthy Food" (curation).

• Performance Gains:
  • vs. Zero-Shot: Agile Deliberation improved F1 scores by 10.5% on average.
  • vs. Automated Decomposition (Modeling Collaborator): Improved F1 by 7.5%.
  • vs. Manual Deliberation: Improved F1 by >3%.
  • Note: Gains were most significant for concepts where users held nuanced interpretations that deviated from generic VLM priors.
• User Experience:
  • Participants reported significantly lower cognitive effort and frustration (p < 0.05) compared to manual deliberation.
  • Users found it easier to identify edge cases and articulate their definitions with the system's structured guidance.
• Transferability: Concept definitions generated by Agile Deliberation (optimized for Gemini 2.5) transferred effectively to other VLMs (Qwen3-VL), with performance gains increasing on more capable models.

5. Significance and Impact

• Democratizing Subjective AI: The framework lowers the barrier for non-experts to encode their specific values and perspectives into vision systems, making high-stakes classification (like content moderation) more accessible and aligned with human intent.
• Bridging the Calibration Gap: By focusing on semantic borderline cases rather than model confidence, the system effectively aligns generative VLMs with human reasoning, which is often uncalibrated.
• Proactive Risk Mitigation: In domains like content moderation, this approach allows practitioners to proactively design classifiers that anticipate nuanced risks and harms, rather than merely reacting to them after deployment.
• Future Direction: The paper suggests that future AI systems for subjective tasks must move beyond static training data to include interactive, deliberative loops that model the evolution of human understanding.