💬 NLP

Models Know Models Best: Evaluation via Model-Preferred Formats

This paper proposes a dynamic format-alignment strategy that leverages a lightweight classifier trained on model-preferred signals to automatically select between symbol-based and cloze-style evaluation formats, thereby resolving performance discrepancies and significantly improving zero-shot accuracy across various LLM benchmarks.

Original authors: Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joonhak Lee, Sungmok Jung, Jongyeon Park, Jaejin Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a test to prove how smart you are. Now, imagine the test comes in two different styles:

The "Multiple Choice" Style: You read a question, and then you have to pick a letter (A, B, C, or D) that matches the answer.
The "Fill-in-the-Blank" Style: You read a sentence that stops abruptly, and you have to complete the sentence with the right words.

For a long time, researchers thought these two styles were just different ways of asking the same question. But this paper, titled "Models Know Models Best," discovered something surprising: Large Language Models (LLMs) don't like both styles equally. In fact, the style you choose can make a model look like a genius or a complete failure, even if the question is exactly the same.

Here is the breakdown of what the authors found, using some simple analogies.

1. The "Wrong Tool for the Job" Problem

The researchers tested 22 different AI models on 8 different types of questions. They found a clear pattern:

The "Symbol" Style (Picking A, B, C, D): This works best for questions that require comparing options. Think of it like a menu where you have to look at a list of dishes and pick the one that fits your diet. The model has to look at all the options side-by-side to make a choice.
The "Cloze" Style (Fill-in-the-blank): This works best for questions that require continuing a story. Think of it like finishing a sentence in a novel. The model is trained to predict what word comes next in a natural flow.

The Problem: When researchers forced a "story-telling" model to pick a letter from a list (Symbol style) for a question that was really about finishing a sentence, the model's score crashed. It was like asking a marathon runner to solve a math equation; they are good at running, but the test format didn't match their strength.

2. Humans Can't Guess the Best Format

The authors first tried to fix this by asking humans to look at a question and decide, "Hey, this one looks like a story, so let's use the Fill-in-the-blank style."

The Result: It didn't work well. Humans are bad at guessing which format an AI will prefer. Sometimes a question looks like a story to a human, but the AI actually prefers the multiple-choice list. Relying on human intuition often made the AI perform worse.

3. The Solution: "Ask the Model What It Wants"

Since humans couldn't guess the right format, the authors asked the models themselves.

They built a tiny, lightweight "traffic cop" (a classifier). This traffic cop looks at a specific question and asks the AI: "If I give you this question as a multiple-choice list, how confident are you? If I give it to you as a fill-in-the-blank, how confident are you?"

Based on the AI's own internal confidence signals, the traffic cop decides which format to use for that specific question.

If the question is about comparing options: The traffic cop says, "Use the Multiple Choice (Symbol) format."
If the question is about finishing a sentence: The traffic cop says, "Use the Fill-in-the-blank (Cloze) format."

4. The Results: Unlocking Hidden Potential

When they used this "Model-Preferred" strategy, the results were dramatic.

For "Story" questions: The AI's performance jumped significantly. It was as if they finally gave the runner the right shoes.
For "Comparison" questions: The performance stayed high or improved slightly.
The Big Takeaway: The paper shows that many AI models are actually much smarter than we thought, but we were often testing them with the wrong "ruler." By switching the ruler to match the specific task, we can see their true capabilities.

Summary

The paper argues that we shouldn't just pick one test format and stick with it for everything. Instead, we should let the AI tell us which format it prefers for each specific problem. By doing this, we stop tripping the models up and start seeing how smart they really are.

In short: The paper proves that the way you ask a question matters just as much as the question itself, and the best way to figure out the right way to ask is to listen to the model's own preferences.

Technical Summary: Models Know Models Best: Evaluation via Model-Preferred Formats

Problem Statement

Multiple-choice question answering (MCQA) is the de facto standard for evaluating Large Language Models (LLMs). However, recent studies indicate that model performance in MCQA is highly sensitive to input formatting, particularly the choice between symbol-based and cloze-style evaluation formats.

Symbol-based: The model selects an option by generating a label (e.g., "A", "B", "C", "D") corresponding to a list of choices.
Cloze-style: The model evaluates the likelihood of the answer text itself when appended to the prompt, often requiring length normalization.

Existing research highlights significant performance discrepancies between these two formats, yet the underlying causes remain poorly understood. Crucially, it is unclear whether these discrepancies are stochastic or systematic, and whether human-designed heuristics can reliably predict the optimal format for a given task. Current evaluation protocols often rely on a single fixed format, potentially underestimating the true capabilities of LLMs due to format-task misalignment.

Methodology

The paper proposes a dynamic format-alignment strategy that selects the optimal evaluation format for each problem instance based on the model's own latent preferences, rather than human intuition.

1. Empirical Baseline Analysis

The authors conducted a systematic comparison across 22 decoder-based LLMs (including Qwen 3, Llama 3.1/3.2, and Mistral families) and eight diverse benchmarks (e.g., MMLU, ARC, HellaSwag, PIQA).

Findings: Performance differences are systematic and task-dependent.
- Symbol-favoring tasks: Tasks requiring explicit comparison of options (e.g., "Which of the following is...?") or containing options like "None of the above" (e.g., ARC, MMLU) perform significantly better with symbol-based formats.
- Cloze-favoring tasks: Tasks requiring natural language continuation (e.g., predicting the next sentence in a narrative) perform significantly better with cloze-style formats (e.g., HellaSwag, PIQA, WinoGrande).
In-Context Learning: While few-shot demonstrations can mitigate format mismatches, they do not fully eliminate the performance gap, and models often fail to reach the baseline of the optimal format even with 10 examples.

2. Classifier-Based Format Selection

To address these inconsistencies, the authors introduce a lightweight encoder-based classifier (fine-tuned DeBERTaV3) that predicts the preferred format (Symbol vs. Cloze) for each problem instance. Two labeling strategies were explored for training this classifier:

Human-Annotated Labels: A subset of data was manually labeled based on syntactic heuristics (e.g., whether the option completes a sentence or requires explicit comparison).
- Result: This approach led to consistent performance degradation, suggesting that human linguistic intuition does not align with the latent inductive biases of LLMs regarding format preference.
Model-Generated Labels: Labels were derived by evaluating the target model (or a set of models) on both formats and selecting the one where the model demonstrated higher correctness or confidence (measured via probability margins).
- Algorithm: For a given instance, if the model answers correctly in only one format, that format is chosen. If correct in both, the format with the higher confidence margin is selected (with a threshold $\delta$ to avoid noise).
- Generalization: To test transferability, a "majority-voting" strategy was employed where labels were aggregated from three distinct mid-sized models (Llama 3.1 8B, Mistral 7B, Qwen3 8B) to train a classifier for unseen models.

Key Contributions

Systematic Characterization of Format Bias: The paper establishes that performance discrepancies in MCQA are not stochastic but vary systematically according to the structural and linguistic properties of the task. These patterns are model-agnostic, emerging consistently across diverse decoder-based LLMs.
Latent Preference Signals: The authors provide evidence that the characteristics governing format preference are implicitly encoded within the model's latent representations. Explicitly eliciting these signals allows for the recovery of performance otherwise lost to suboptimal formatting.
Dynamic Format-Alignment Strategy: The proposed method leverages model-generated signals to dynamically select the optimal evaluation format per instance. This approach bridges the gap between model capability and task characteristics without requiring training-free heuristics.

Results

The evaluation results demonstrate that the model-generated classifier significantly outperforms both fixed-format baselines and human-annotated classifiers.

Performance Gains: The proposed method yields substantial and consistent improvements in zero-shot accuracy across reasoning and knowledge-intensive benchmarks.
- On cloze-favoring tasks (e.g., HellaSwag, PIQA, WinoGrande), the approach often surpasses the baseline of the "preferred" format, recovering significant performance (e.g., +16.7% on WinoGrande for Llama 3.2 1B, +28.3% for Qwen3 0.6B).
- On symbol-favoring tasks (e.g., ARC, MMLU), the method maintains parity with the original preferred format, avoiding the degradation seen with human-annotated classifiers.
Generalization: The "majority-labeled" approach, which aggregates signals from multiple models, achieves gains comparable to "self-labeled" classifiers. This indicates that format preferences are not unique artifacts of specific models but reflect shared, task-dependent tendencies inherent to decoder-based LLMs.
Human vs. Model Heuristics: Classifiers trained on human-annotated labels consistently degraded performance, confirming that human intuition is an unreliable proxy for model format preferences.

Significance

The paper argues that fixed-format evaluation practices currently underestimate LLM capabilities. By demonstrating that models possess latent, task-dependent biases regarding evaluation formats, the authors emphasize the importance of aligning evaluation protocols with these inductive biases.

The significance of this work lies in its methodological contribution to LLM evaluation:

It challenges the assumption that human-designed heuristics are sufficient for determining optimal evaluation strategies.
It introduces a training-free, instance-level mechanism to recover latent model capabilities that are obscured by format mismatches.
It suggests that future evaluations should consider how models internally represent and respond to different interaction formats to provide a more accurate and insightful assessment of model performance.

The authors note that this research is primarily methodological, aiming to improve the reliability and interpretability of model evaluation, and does not directly raise new societal or ethical concerns beyond those inherent to LLM evaluation itself.