When Models Know More Than They Say: Probing Analogical Reasoning in LLMs

1. Problem Statement

The paper addresses a critical gap in evaluating Large Language Models (LLMs): the discrepancy between a model's internal representations (what it "knows") and its prompted behavior (what it "says" or can do). While LLMs perform well on tasks where surface-level and structural cues align, they often struggle with analogical reasoning in narratives where the analogy is latent and requires abstraction beyond surface similarity.

The core research question is: Do LLMs internalize the typological structures and narrative schemas necessary for analogical reasoning, or are they merely leveraging surface-level correlations? Specifically, the authors investigate whether information decodable from internal model states (via probing) is accessible through standard prompting techniques.

2. Methodology

2.1 The NARB Benchmark

The authors introduce NARB (Narrative Analogical Reasoning Benchmark), a suite of tasks designed to test analogical reasoning in literary texts. The benchmark consists of two distinct tasks:

1. Narrative Parallelism: Determining if two complete narratives share a systematic structural correspondence (e.g., a shared proverb or moral) despite differing in setting, characters, or vocabulary. This tests high-level abstraction.
   • Dataset: ARN (Analogical Reasoning over Narratives), derived from ePiC stories.
2. Rhetorical Parallelism: Identifying localized stylistic and semantic symmetry within a document (e.g., syntactic inversion, antithesis, or chiasmus). This tests fine-grained form-meaning correspondence.
   • Dataset: ASP (Augustinian Sermon Parallelism), containing annotated Latin sermons.

2.2 Experimental Design

The study employs a dual-method approach to evaluate model capabilities:

• Probing (Internal State Analysis):
  • The authors extract hidden state activations from decoder-only Transformer models (LLaMA 3 family: 1B, 3B, 8B parameters) across all layers.
  • They train lightweight, low-capacity classifiers (Linear/Logistic Regression and shallow MLPs) on these representations to predict parallelism.
  • They analyze layer-wise progression to determine where specific information is encoded.
• Prompting (Behavioral Analysis):
  • Models are prompted to rank candidate narratives or spans based on their parallelism to an anchor.
  • Both open-source (LLaMA 3) and closed-source (GPT-5.2, Claude Opus 4.5) models are evaluated.
  • The setup uses a ranking formulation (Mean Average Precision - MAP) rather than binary classification to capture nuance.

2.3 Evaluation Metrics

• Primary Metric: Mean Average Precision (MAP) for ranking tasks.
• Secondary Metrics: Mean Reciprocal Rank (MRR), Pairwise Accuracy, and F1/AUROC for auxiliary tasks.
• Baselines: Cosine similarity of embeddings, distance-only baselines (for rhetorical tasks), and non-LLM linguistic methods (POS, N-grams, dependency trees).

3. Key Contributions

1. NARB Benchmark: A novel evaluation framework distinguishing between narrative (macro) and rhetorical (micro) analogical reasoning.
2. Probing vs. Prompting Asymmetry: The paper provides empirical evidence of a task-dependent dissociation between internal knowledge and external behavior.
3. Diagnostic Insights: It demonstrates that for certain tasks, models possess high-fidelity internal representations that they fail to utilize when prompted, challenging the assumption that prompting fully reveals model capabilities.

4. Key Results

4.1 Rhetorical Parallelism (The "Hidden Knowledge" Case)

• Probing Performance: Exceptionally high. Open-source models (LLaMA-3-1B) achieved a MAP of ~0.93 when probed.
• Prompting Performance: Extremely low. The same model achieved only MAP ~0.18 when prompted.
• Analysis: The internal representations contain strong signals for rhetorical structure (linearly decodable), but the model fails to "recruit" this knowledge during instruction following.
• Locality Dependence: A distance-only baseline (MAP 0.98) outperformed embedding-based methods, suggesting rhetorical parallelism in the dataset relies heavily on spatial proximity (locality) rather than deep semantic abstraction.

4.2 Narrative Parallelism (The "Weak Encoding" Case)

• Probing Performance: Moderate. Open-source models achieved MAP ~0.35.
• Prompting Performance: Converged with probing. Open-source models also achieved MAP ~0.35.
• Analysis: Narrative analogy is genuinely difficult; the models do not robustly encode the high-level abstract schemas required, and thus prompting cannot access what isn't there.
• Closed-Source Gap: Closed-source models (GPT-5.2, Claude Opus) significantly outperformed open-source models in both probing and prompting (MAP ~0.82), suggesting better abstraction capabilities.

4.3 Layer-wise Analysis

• Rhetorical: Information is concentrated in later layers (8–15), with a sharp increase in performance around layer 9.
• Narrative: Information is distributed relatively evenly across layers, suggesting it relies on integrated semantic representations rather than a specific structural layer.

5. Significance and Implications

• Limitations of Prompting: The study argues that relying solely on prompting to evaluate LLMs leads to an underestimation of model capabilities, particularly for structured tasks like rhetorical analysis. Models may "know" the answer but fail to "say" it.
• Interpretability Validity: The results suggest that probing is a valid diagnostic tool, but its agreement with prompting is task-dependent.
  • Disagreement (Rhetorical): Indicates a failure in the model's ability to access encoded knowledge via natural language instructions.
  • Agreement (Narrative): Indicates a genuine lack of capability/encoding.
• Architectural Insights: The findings highlight that while LLMs can encode complex structural patterns (like rhetorical symmetry) in their internal states, the mechanism for translating these states into coherent, instructed behavior remains a bottleneck, especially in smaller, open-source models.

In conclusion, the paper posits that analogical reasoning in LLMs is not a monolithic capability. It is a spectrum where some forms (rhetorical) are well-encoded but inaccessible, while others (narrative) are poorly encoded and inaccessible. This necessitates a combined evaluation strategy using both probing and prompting to fully understand model intelligence.