Does Slightly Mean Somewhat? Measuring Vague Intensity Words in LLM Numeric Actions

Technical Summary: Measuring Vague Intensity Words in LLM Numeric Actions

Problem Statement
As language models (LLMs) are increasingly embedded in agents and natural-language control systems, they must translate vague human instructions (e.g., "slightly increase," "drastically reduce") into precise numeric actions with downstream consequences. While prior research has established that humans preserve ordinal rankings in vague expressions and that LLMs can be sensitive to wording, there is a gap in understanding how models map these intensity modifiers to concrete numeric outputs in a controlled environment. Specifically, it remains under-measured whether models preserve the hypothesized ordinal structure of intensity words when forced to select a single numeric value, how context (system state) influences this mapping, and how models behave near operational feasibility boundaries.

Methodology
The study employs a synthetic, constrained resource-allocation environment to isolate the translation from language to action.

• Experimental Setup: The model (Claude Haiku) receives natural-language instructions to increase a task allocation using one of 10 specific intensity words. A deterministic backend converts the model's numeric tool call into a measurable outcome.
• Word Scale: Ten English degree modifiers were selected to span six hypothesized tiers based on the Quirk et al. (1985) taxonomy, ranging from weak diminishers (slightly, marginally) to maximizers (drastically, dramatically).
• Conditions:
  1. No-Context: Instructions lack current system state (e.g., "spend [WORD] more time").
  2. Context-Conditioned: Instructions include the starting allocation (e.g., "Current allocation is X%. I would like to [WORD] increase"). Ten baseline values (0% to 89%) were tested, including round numbers and irregular values.
  3. Controls: Exact-number prompts validated the harness.
• Parameters: Experiments were run at temperatures $T=0.0$ and $T=0.7$, with 30 runs per cell (totaling 6,620 runs).
• Analysis: The study utilizes non-parametric statistics, including Spearman's $\rho$ for ordinal correlation and Kruskal-Wallis tests with effect sizes ($\varepsilon^2$) to determine whether word choice or starting allocation explains more variance in the output.

Key Results
Three primary patterns emerged across the experiments:

1. Compression of Intensity: In the no-context condition, the model compresses 10 distinct intensity words into only 5 distinct median outputs. The four lowest-tier words (slightly, marginally, somewhat, mildly) all map to a median allocation of 0.50 with zero variance. Stronger words break into higher regimes (0.60–0.70). While a strong ordinal correlation exists ($\rho = 0.845$), fine-grained lexical distinctions are collapsed.
2. State Dominance: When context is provided, the starting allocation dominates word choice. Grouping by baseline explains roughly 10 times more rank-based variance than grouping by word ($\varepsilon^2_{baseline} = 0.782$ vs. $\varepsilon^2_{word} = 0.079$). As the system approaches capacity, the differentiation between words collapses to zero; at a 75% baseline, all 10 words produce the same median output.
3. Discontinuous Boundary Behavior: Near feasibility limits (89% starting allocation), the model exhibits three distinct behavioral modes dependent on the word used:
   • Hedge: Weak words make small adjustments.
   • Abstain: Strong words (e.g., considerably, substantially) often refuse to act entirely, citing constraints.
   • Push: The word drastically behaves uniquely, pushing the allocation to the local ceiling (0.912) rather than abstaining.
     These patterns persist across temperatures; increasing $T$ to 0.7 broadens distributions but does not restore granular ordinal distinctions.

Key Contributions

1. Methodology: A controlled instrument for measuring how LLMs map vague intensity words to numeric actions, separating the language-to-action translation from downstream system noise.
2. Evidence of Compression: Documentation that models preserve coarse intensity regimes but collapse fine-grained lexical distinctions into identical numeric outputs.
3. Context Dependence: Quantitative evidence that current system state dominates word choice in context-conditioned settings, undermining the reliability of vague instructions as context-invariant control signals.
4. Boundary Discovery: Identification of word-dependent abstention and categorical mode switching (hedge, act, abstain) near operational limits, revealing discontinuities in action policies that are not visible from language alone.

Significance and Claims
The paper posits that vague instructions should not be assumed to map to stable, scalar actions in LLM-mediated interfaces. The findings suggest that the model's operational vocabulary is coarser than surface language implies and that the meaning of an intensity word is a joint function of the word and the system state, not a fixed value.

The study frames these findings as a specific alignment problem: whether a model-mediated interface preserves the operator's intended semantics. The results indicate a mixed outcome: while broad intensity regimes are captured, fine distinctions are lost, and behavior becomes discontinuous near boundaries. The author explicitly notes that the study does not claim the model is "worse" than humans, as human interpretation of these words may also be ambiguous; rather, the study documents the specific behavioral patterns of the tested model in this domain. The paper concludes that these patterns are specific to the tested model and domain, with future work planned to include human baselines, cross-model comparisons, and denser experimental grids.