Technical Summary: A Symmetry-Based Framework for Measuring LLM Preferences Under Conflicting Specifications

Problem Statement

Large language models (LLMs) increasingly operate in environments where multiple contextual specifications define a task. These specifications often take diverse forms—natural language instructions, formal constraints, executable tests, or input–output examples—and may be inconsistent or directly conflicting. While prior research has examined how models detect or reconcile conflicts among information sources (e.g., in retrieval-augmented generation or fact-checking), there is a lack of controllable, attributable methods for systematically analyzing how models prioritize different forms of task specification when they support incompatible behaviors. Specifically, it remains unclear whether LLMs exhibit systematic preferences for certain representation types (e.g., formal vs. natural language) and how these preferences vary across models and task domains.

Methodology

The authors propose a symmetry-based experimental framework designed to measure model preferences under conflicting specifications by treating the problem as a selection process over competing executable mappings.

Core Framework

1. Executable Mappings: A task is defined by a mapping $f: X \to Y$. Conflicting specifications are constructed as pairs of mappings ($f_1, f_2$) that produce distinct, attributable outputs for a discriminative query $q^\star$ (where $f_1(q^\star) \neq f_2(q^\star)$).
2. Attribution: Because each specification induces a concrete output, the model's response can be directly attributed to one mapping, the other, or neither. This transforms an implicit decision into an observable variable.
3. Symmetry-Based Counterbalancing: To isolate representation preferences from confounding factors (such as presentation order or specific content biases), the framework employs symmetry-complete configurations. For any comparison between representation types $A$ and $B$, four trials are generated per conflict instance:
   • $(A(f_1), B(f_2))$, $(B(f_1), A(f_2))$, $(A(f_2), B(f_1))$, $(B(f_2), A(f_1))$.
     This design counterbalances the assignment of representation types to underlying mappings and the order of presentation, allowing preference estimation through aggregation that marginalizes out non-target factors.

Experimental Setup

The framework was instantiated across three domains:

1. Mathematical Benchmark: A controlled dataset of 550 conflict instances across 11 function families (e.g., algebraic, graph, matrix operations). Four representation types were compared: Pure Natural Language (NL), Formal Language (Form), Naturalized Formal Language (NatF), and Input–Output Examples (Ex).
2. Boolean Algebra: Conflicts between symbolic formulas and truth tables.
3. Code Generation: Conflicts between natural language documentation and executable unit tests (derived from the MBPP dataset).
4. Clinical Rules: Conflicts involving renal dosing thresholds and pediatric age indications derived from openFDA labels, testing conflicts between genuine rules and schema-preserving perturbations.

Key Results

1. Systematic Representation Preferences (Mathematical Benchmark)

Across multiple models (including DeepSeek, GPT, and Llama families), the study identified a consistent preference ordering for resolving mathematical conflicts:
$$\text{Formal} \approx \text{Naturalized Formal} > \text{Pure Natural Language} > \text{Input–Output Examples}$$

• Formal vs. Natural Language: Models selected formal specifications over pure natural language in ~87% of attributable responses.
• Formal vs. Examples: The preference for formal specifications over examples was even stronger (~93%).
• Naturalized Formal: Despite being expressed in natural language, naturalized formal specifications behaved similarly to pure formal specifications, suggesting that the presence of explicit mathematical structure is the dominant factor, not the surface syntax.
• Model Variance: While the overall hierarchy held, the relative strength of preferences varied. For instance, the preference for formal over naturalized formal was not uniform; some models (e.g., GPT-5-nano) favored naturalized formal, while others (e.g., DeepSeek-V4-Flash) favored pure formal.

2. The Role of Examples

The competitiveness of input–output examples was found to be highly dependent on model capability and task structure:

• Capability Dependence: Higher-capability model variants generally selected example-supported mappings more frequently than lower-capability variants.
• Task Dependence: Examples were most competitive against pure natural language (selection rates up to ~55% for some models) but significantly less competitive against formal or naturalized formal specifications (selection rates often <10%).
• Structure Sensitivity: Examples struggled when the competing specification exposed explicit mathematical structure, as the finite demonstrations were insufficient to recover the rule compared to the explicit constraints of formal notation.

3. Heterogeneous Domain Findings

• Boolean Algebra: Models exhibited a strong preference for compact, compositional symbolic representations (formulas and gate netlists) over exhaustive truth tables. However, the preference between formulas and gate netlists was model-dependent, suggesting no universal hierarchy for structurally similar representations.
• Code Generation: A sharp divergence was observed based on model capability. Lower-scoring models generally favored natural language documentation, while higher-scoring models strongly favored executable tests (up to 97% selection for top models).
• Clinical Rules: In clinical settings, the representation hierarchy did not transfer cleanly. Instead, a domain-specific bias emerged: models consistently preferred the more restrictive rule (52%–87% selection), regardless of whether that rule was the "genuine" label or a perturbation. This indicates that in high-stakes domains, safety/restrictiveness biases can override representation preferences.

4. Robustness and Abstention

• Presentation Order: While a secondary effect, presentation order had a minimal impact compared to representation type, except when competing specifications had similar strength (e.g., Formal vs. Naturalized Formal).
• Abstention: When explicitly allowed to answer "uncertain," models (particularly GPT-5-nano) utilized abstention as a conflict-handling strategy, with rates varying significantly by representation pair (e.g., ~64% abstention for Formal vs. Naturalized Formal conflicts in GPT-5-nano).

Significance and Claims

The paper claims to establish conflict resolution in LLMs as a controlled, executable, and symmetry-counterbalanced measurement problem.

• Attributable Measurement: By using executable ground truth and symmetry-based counterbalancing, the framework enables the direct attribution of model outputs to specific specifications, moving beyond qualitative analysis of "hallucination" or "reconciliation."
• Structured but Context-Dependent: The results characterize LLM conflict resolution not as random noise, but as a structured selection process. However, this structure is context-dependent: while a general preference for explicit structure exists in mathematical tasks, domain-specific factors (like restrictiveness in clinical rules or test precision in code generation) can reshape or override these preferences.
• Unified Framework: The work provides a unified experimental approach to analyze how models prioritize different forms of specification, demonstrating that these preferences are not universal but vary systematically across models, task families, and domains.

The authors conclude that understanding these implicit hierarchies is essential for designing reliable LLM-based systems, as the "winner" in a conflict is determined by a complex interplay of representation type, model capability, and domain-specific constraints.