Funny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs

Technical Summary: Funny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs

Problem Statement
Large Language Models (LLMs) possess strong generative capabilities, yet many applications require explicit, fine-grained control over specific textual concepts such as humor, persuasiveness, or formality. While prior work has explored control via prompting, representation engineering, and style transfer, these approaches generally offer coarse or single-attribute control. Existing benchmarks, such as SCTG, assess calibration for one attribute at a time but fail to evaluate how models behave when two distinct concepts are controlled simultaneously. The central problem addressed is the lack of systematic evaluation for fine-grained multi-concept controllability and the unknown extent to which models can disentangle and compose independent stylistic dimensions.

Methodology
The authors introduce a systematic evaluation framework designed to assess fine-grained controllability in both single- and dual-concept settings.

• Framework Definition: The task is defined as generating text $y$ given a context $x$ and target concepts $C_a, C_b$ with desired discrete levels $\ell \in \{0, \dots, 4\}$. The framework evaluates:
  1. Single-concept control: Generating outputs across levels for one concept.
  2. Dual-concept control: Generating outputs where one concept is the target (varying levels) while a secondary concept is held fixed at a specific level or sampled randomly.
• Evaluation Protocol:
  • Models: The study evaluates medium-sized, instruction-tuned LLMs (7B–14B parameters), specifically Llama 3.2-11B, Gemma 3-12B, and Qwen3-14B.
  • Concepts: Six linguistically distinct concepts are tested: humor, persuasiveness, clarity, politeness, assertiveness, and formality. Three pairs are selected based on theoretical and empirical evidence of their distinctness: humor–persuasiveness, clarity–politeness, and assertiveness–formality.
  • Tasks: Experiments cover argument generation, story generation, and structured text generation.
  • Metric: A stronger LLM (GPT-4.1) acts as a judge, performing pairwise comparisons of generated outputs to determine which exhibits more of the target concept. The alignment between the intended levels and the judge's empirical ranking is measured using Spearman rank correlation ($\rho$). Fisher z-transformations are applied for statistical aggregation and hypothesis testing.
• Control Method: The study focuses on prompt-based control, utilizing structured prompt templates that explicitly encode desired concept intensities.

Key Results
The experiments reveal a significant gap between single-concept and dual-concept performance:

1. Performance Degradation in Dual-Concept Settings: While models generally achieve strong fine-grained calibration for individual concepts (high Spearman correlations), performance drops sharply when a secondary concept is introduced. This decline occurs even for concept pairs that are intuitively orthogonal (e.g., clarity vs. humor).
2. Compositionality Failure: The results suggest that concept dimensions in current LLMs are entangled in ways that resist naive composition. Even when concepts should be independent, the presence of a secondary control signal compromises the model's ability to maintain the intended level of the primary concept.
3. Model and Task Variance:
   • Model Size/Architecture: Qwen-14B and Gemma-12B consistently outperform Llama-11B across all settings, suggesting that larger or more instruction-tuned models better preserve disentanglement between stylistic dimensions.
   • Task Context: Narrative generation allows for more flexible style variation, whereas argumentative and structured contexts amplify conflicts between stylistic goals, leading to more pronounced performance drops.
4. Statistical Significance: Paired t-tests on Fisher-transformed correlations confirm that the introduction of a secondary concept significantly harms controllability for most concept pairs and models (e.g., $p < 0.05$ for many comparisons).

Key Contributions

1. Evaluation Framework: The paper provides a principled, model- and method-agnostic framework for measuring fine-grained multi-concept controllability, filling a gap left by single-attribute benchmarks.
2. Empirical Evidence of Limitations: It offers systematic evidence that current LLMs struggle with the compositionality of stylistic control, even when concepts are theoretically independent.
3. Baseline for Future Research: By establishing clear metrics and identifying common failure modes (specifically the drop in performance during dual-concept control), the work sets a foundation for developing more robust methods for interpretable, multidimensional stylistic control.

Significance
The paper claims that while prompting offers a widely accessible method for single-concept control, it reveals a fundamental limitation: models struggle to jointly coordinate multiple stylistic dimensions. The authors posit that this work establishes a necessary foundation for future research into interpretable and compositional concept control. By quantifying the gap between intended and realized control in multi-concept scenarios, the framework aims to encourage the development of methods capable of precise, fine-grained, and disentangled control across multiple stylistic axes in language models. The authors remain modest, noting that their conclusions do not assume conceptual separability in the model's internal representations but rather focus on the user-facing ability to specify and achieve separate levels for distinct concepts.