Grading the Unspoken: Evaluating Tacit Reasoning in Quantum Field Theory and String Theory with LLMs

This paper introduces a five-level expert-curated evaluation framework to demonstrate that while large language models excel at explicit derivations in quantum field theory and string theory, they systematically fail when tasks require reconstructing tacit reasoning or maintaining global conceptual consistency, thereby revealing the limitations of current evaluation paradigms for highly abstract theoretical physics.

The Gist

The Big Idea: Can AI "Get" the Unspoken Rules?

Imagine you are trying to teach a brilliant but very literal robot how to cook a complex dish, like a soufflé. You give the robot a recipe that says: "Whisk the eggs until fluffy, add the flour, and bake."

The robot follows the instructions perfectly. It whisks, adds flour, and bakes. But the soufflé collapses. Why? Because the recipe didn't mention the tacit knowledge: the fact that you need to fold in the egg whites gently, or that the oven needs to be preheated to a specific temperature, or that you can't open the door halfway through.

In the world of high-level physics (Quantum Field Theory and String Theory), experts work the same way. They often skip steps in their explanations because, to them, those steps are "obvious." They assume everyone else in the club knows the unspoken rules of the game.

This paper asks a tough question: Can Large Language Models (LLMs) like the ones we use today figure out these missing, unspoken steps, or do they just look good on the surface?

The Experiment: The "Missing Link" Test

The researchers created a special "exam" for AI. Instead of asking the AI to solve a math problem and check if the final number is right, they asked it to explain why certain deep physics concepts work, specifically focusing on the parts experts usually skip.

They built a 12-question quiz covering the most abstract parts of modern physics. To grade the answers, they didn't just use a "Pass/Fail" system. They invented a 5-Star Hotel Grading System:

• ⭐ (Level 0): The AI got the final answer right. (Like a robot that serves a burnt cake but says, "Here is your cake.")
• ⭐⭐ (Level 1): The AI knows the right vocabulary and names the key concepts. (It knows the cake needs "flour" and "eggs.")
• ⭐⭐⭐ (Level 2): The AI connects the dots. It explains how the ingredients lead to the result. (It explains the recipe steps.)
• ⭐⭐⭐⭐ (Level 3): The Magic Level. The AI fills in the missing steps that the experts skipped. It explains the "why" behind the "how." (It explains why you must fold the eggs gently, even though the recipe didn't say so.)
• ⭐⭐⭐⭐⭐ (Level 4): The AI goes above and beyond, offering new insights or real-world examples. (It suggests a better way to bake the cake for a specific diet.)

The Results: The "Smart but Literal" Problem

The results were fascinating and a bit worrying for the future of AI research.

1. The "Surface" is Great:
Almost every AI model got Level 0, 1, and 2 almost perfectly. They could recite the facts, name the theories, and write down the standard formulas. If you just asked, "What is the answer?", they sounded like Nobel Prize winners.

2. The "Deep" is Broken:
When the test required Level 3 (filling in the missing, unspoken logic), the scores crashed.

• The Problem: The AI models are like actors who have memorized the script but don't understand the character's motivation. They can say the lines, but they can't improvise when the script has a gap.
• The "Conceptual Hinge" Failure: The hardest questions were the ones where the AI had to realize, "Wait, I'm looking at this problem from the wrong angle. I need to change my whole perspective to make sense of it." The AI models mostly failed here. They tried to force the answer using the wrong lens, rather than stepping back and realizing the lens itself was wrong.

The "Hint" Experiment: A Lightbulb Moment

To see if the AI was actually "dumb" or just "confused," the researchers tried a trick. They took a question where the AI failed and added a tiny hint: "By the way, make sure you distinguish between these two similar-sounding words."

The Result: The AI's performance skyrocketed.

• What this means: The AI actually knew the answer. It just couldn't figure out which tool to use to solve the puzzle on its own. It's like a student who knows the math but doesn't realize they need to use the quadratic formula until the teacher points it out.

The Metaphor: The Tourist vs. The Local

Think of the AI as a Tourist with a perfect guidebook, and a human expert as a Local.

• The Tourist (AI): Can recite the guidebook perfectly. "The museum is at 5th and Main. It opens at 9." (Level 0-2).
• The Local (Human): Knows the unspoken rules. "Don't go to the museum on Tuesdays because the janitor locks the back door early, and if you want to see the real art, you have to ask the guard for the key." (Level 3-4).

The paper shows that current AI is a very well-read Tourist. It can tell you where the museum is, but it can't navigate the hidden backdoors or the unspoken social rules of the city.

Why Does This Matter?

This paper suggests that while AI is amazing at reproducing what humans have already written, it is currently terrible at reconstructing the deep, invisible logic that experts use to think.

In fields like physics, where the "real" work happens in the gaps between the written words, current AI models are hitting a wall. They can't yet act as true research partners because they can't "think outside the box" unless someone explicitly tells them where the box is.

The Bottom Line: AI is great at being a librarian who can find any book. But it's not yet a researcher who can write a new book by understanding the unwritten rules of the genre. To get there, we need to teach AI not just what to say, but how to think when the instructions are missing.

Technical Summary

1. Problem Statement

Current Large Language Model (LLM) benchmarks in physics and mathematics primarily evaluate explicit reasoning (step-by-step derivations) or final answer correctness. However, in highly abstract theoretical fields like Quantum Field Theory (QFT) and String Theory, expert knowledge is often tacit.

• The Gap: Standard literature routinely omits intermediate steps considered "self-evident" to experts or relies on implicit structural constraints (e.g., global consistency, topological obstructions) rather than explicit local derivations.
• The Challenge: Standard evaluation metrics (answer matching) fail to distinguish between a model that correctly reproduces a result via pattern matching and one that has genuinely reconstructed the underlying implicit reasoning structures. The paper asks: Can LLMs reconstruct the omitted reasoning steps and implicit constraints that define expert competence in theoretical physics?

2. Methodology

A. Dataset Construction

The authors constructed a compact, expert-curated dataset of 12 questions spanning core areas of QFT and String Theory (e.g., Chern-Simons theory, D-branes, anomalies, supersymmetry).

• Design: Each question targets a standard statement in the literature where the intermediate reasoning is typically compressed or omitted.
• Taxonomy: The questions are mapped onto a 2D Reasoning Geometry defined by two axes:
  1. Mode of Inference: Mechanism-driven (expanding local derivations) vs. Consistency-driven (identifying global constraints).
  2. Conceptual Organization: Within-frame (fixed conceptual structure) vs. Cross-frame (requiring a shift in representational framing).
  • This creates four regimes: Local Derivation, Integration, Constraint-based, and Conceptual Hinge.

B. Evaluation Rubric (5-Level Grading)

To capture the nuance of tacit reasoning, the authors introduced a hierarchical 5-level scoring rubric:

• Level 0 (Statement Correctness): The final claim is factually correct (recall-based).
• Level 1 (Key Concept Awareness): The response identifies relevant concepts/theorems but lacks causal explanation.
• Level 2 (Reasoning Chain Presence): A causal structure links concepts, but intermediate steps may still be missing.
• Level 3 (Tacit Step Reconstruction): The model explicitly reconstructs steps usually omitted in literature (e.g., topological obstructions, infrared divergences) and integrates them into the argument.
• Level 4 (Enrichment): The response demonstrates broader conceptual awareness (validity regimes, alternative perspectives, physical examples).

C. Experimental Setup

• Models Evaluated: A diverse set of contemporary LLMs, including Gemini (2.5-flash, 3.1-pro), GPT (4.1, 5.2), Deepseek (V3.2), Kimi, Qwen, Minimax, and Nemotron.
• Perturbation Study: A specific case study (Question 11) tested "Conceptual Hinge" tasks by modifying prompts to explicitly hint at the required structural distinction (e.g., distinguishing types of anomalies) to see if models could utilize the hint.

3. Key Results

A. Performance Stratification

• Levels 0–2: Models achieved near-ceiling performance across all models. They can recall facts and identify concepts.
• Level 3 (The Bottleneck): A sharp qualitative drop occurs here. While top models (e.g., Gemini-3.1-pro) maintained high scores (~0.92), most others degraded significantly (0.17–0.50). This indicates that reconstructing omitted intermediate steps is a major failure point.
• Level 4: Scores were generally low, as this requires advanced conceptual synthesis beyond standard training data.

B. Reasoning Geometry Analysis

Performance varied drastically based on the reasoning regime:

1. Local Derivation (Mechanism/Within-frame): High performance. Models handle explicit, sequential derivations well.
2. Integration (Mechanism/Cross-frame): Moderate degradation. Difficulty arises in synthesizing distinct reasoning threads.
3. Constraint-based (Consistency/Within-frame): Early degradation. Models struggle to enforce global consistency conditions throughout the derivation.
4. Conceptual Hinge (Consistency/Cross-frame): Catastrophic failure. Performance collapsed at Level 3 for most models. These tasks require identifying a latent structural distinction (e.g., re-framing the problem) before derivation can proceed. Models failed to initiate the correct representational frame.

C. Failure Mechanism & Prompt Sensitivity

• Root Cause: The failure is not a lack of technical knowledge but an instability in representation selection. Models operate in a "forward-expansion" regime within a fixed frame and fail to autonomously reorganize the conceptual space when global constraints are violated.
• Prompt Sensitivity: In the Q11 perturbation study, providing explicit hints about the required structural distinction (e.g., "distinguish different notions of anomaly") allowed several models to jump from Level 0/1 to Level 3/4. This confirms that the reasoning capability exists but is not robustly triggered without explicit scaffolding.

4. Key Contributions

1. Novel Dataset: A targeted, expert-curated dataset of 12 questions specifically designed to probe tacit reasoning in high-energy theoretical physics, moving beyond standard "textbook" problems.
2. Evaluation Framework: A 5-level grading rubric that decomposes correctness into dimensions of reasoning reconstruction, specifically isolating "Tacit Step Reconstruction" (Level 3) as a critical metric.
3. Reasoning Geometry: A conceptual framework (Mechanism vs. Consistency; Within-frame vs. Cross-frame) that successfully categorizes reasoning tasks and predicts model failure modes.
4. Empirical Insight: Evidence that current LLMs are "mechanism-driven" but lack the ability to perform "consistency-driven" re-framing, identifying representation selection as the primary epistemic bottleneck for AI in theoretical physics.

5. Significance

• Epistemic Limits: The paper argues that highly abstract theoretical physics serves as a uniquely sensitive lens for exposing the limits of current AI evaluation paradigms. Standard benchmarks miss the "unspoken" logic that defines expert competence.
• Research Implications: For AI to assist in theoretical physics research, it must move beyond pattern matching and explicit derivation to autonomous conceptual re-framing. The current generation of models is effective at "doing the math" but fails at "knowing which math to do" when implicit constraints are involved.
• Future Directions: The findings suggest that future evaluation and model development must focus on consistency-driven inference and the ability to detect and resolve conceptual tensions, rather than just scaling up context windows or training data.