LLMs with in-context learning for Algorithmic Theoretical Physics

This paper demonstrates that a frontier Large Language Model (Claude) interfaced with a computer algebra system (Maple) and enhanced with in-context learning via worked examples can reliably execute complex algorithmic computations in theoretical physics, specifically for cosmological perturbations in modified gravity theories, while also identifying current limitations and improvement strategies.

The Gist

The Big Idea: The "Super-Intern" with a Calculator

Imagine a theoretical physicist as a master chef. They are brilliant at inventing new recipes (theories) and understanding the deep flavors of the universe. However, a huge part of their job involves tedious, repetitive chopping and measuring—calculating how tiny ripples in space-time behave. This is what the authors call "algorithmic computations." It's not about inventing a new theory; it's about following a complex, step-by-step recipe to get a specific number.

The authors asked: Can we hire a "Super-Intern" (a Large Language Model or LLM) to do this chopping and measuring for us?

They found that if you just give the intern a general job description, they get confused. But if you give them a cookbook of past successful recipes (called "worked examples") and a powerful calculator (a Computer Algebra System called Maple), the intern can actually do the job quite well.

The Experiment: Teaching a Robot to Do Physics Homework

The researchers set up a test where they asked an AI (specifically, a model called Claude) to solve nine different physics problems. These problems involved figuring out the "degrees of freedom" in the universe—basically, counting how many independent ways a theory of gravity can wiggle and vibrate.

Think of it like this: If you shake a guitar string, it vibrates in a specific way. If you shake a drum, it vibrates differently. The AI's job was to look at a complex mathematical description of a "universe-drum" and tell the physicist exactly how many distinct vibration modes it has.

To help the AI, the researchers tried four different ways of giving it instructions (called "contexts"):

1. The "10-Example Cookbook" (10ex): They gave the AI a thick book containing 10 fully solved examples of similar problems, complete with the math code used to solve them.
2. The "3-Example Sampler" (3broad): They gave the AI just three examples, chosen to be very different from each other.
3. The "Tailored 3-Example" (3tailored): They gave the AI three examples, but they specifically tweaked two of them to show the AI how to avoid the specific mistakes it had made in the first two tests.
4. The "Recipe Card" (instruction): They gave the AI a written list of rules on how to solve the problem, but no actual examples or code.

The Results: What Worked and What Didn't

1. The "Recipe Card" Failed Miserably
When the AI was just given a list of rules (Context 4), it struggled. It tried to do the math in its head without using the calculator, got lost, and gave up. It's like giving a chef a list of ingredients and saying "make a cake," but not showing them how to mix the batter. The AI didn't know how to use the tools.

2. The "Cookbook" Worked Best
When the AI was given the 10 examples (Context 1), it did a great job. It learned by imitation. It saw how the previous problems were solved and copied the pattern. It solved 5 out of 9 problems correctly.

3. The "Tailored" Examples Were the Magic Bullet
The best results came from the Tailored 3-Example set (Context 3). The researchers noticed the AI kept making the same specific mistakes (like forgetting to handle a certain type of math term). They created a new example specifically designed to show the AI how not to make that mistake.

• The Result: This approach solved the most problems (7 out of 9) and did so much faster. It's like a teacher noticing a student keeps forgetting to carry the "1" in addition, so they give them a specific practice problem just for that one error.

The "Thinking" Mode Surprise

The researchers also tried turning on a "Thinking Mode" for the AI, which allows it to pause and "think" before answering (like a human taking a deep breath). Surprisingly, this didn't help much. The AI didn't use this extra time to solve harder problems; it just took longer to make the same mistakes.

The "Human-in-the-Loop" Reality Check

The paper emphasizes that this AI is not a replacement for a physicist. It's more like a very fast, very literal intern.

• It gets stuck: Sometimes the AI tries to solve a problem, realizes it's going in circles, and restarts its calculator session (like a student erasing their whole page and starting over).
• It needs supervision: The AI sometimes misinterprets the problem (e.g., thinking a "flat" universe means something different than it does). A human expert is needed to catch these errors.
• It's good at the boring stuff: The AI is excellent at the repetitive, mechanical math that physicists hate doing, but it still needs a human to check the final answer.

The Bottom Line

The paper concludes that Large Language Models can be powerful tools for theoretical physics, but only if you teach them correctly. You can't just ask them to "be smart." You have to show them examples of how to do the work, specifically pointing out the tricky parts where they usually fail.

When equipped with a calculator and a good set of "worked examples," an AI can solve complex physics problems with the competence of a first-year graduate student, freeing up human physicists to focus on the creative parts of their work.

Technical Summary

Technical Summary: LLMs with In-Context Learning for Algorithmic Theoretical Physics

Problem Statement
Theoretical physics involves a spectrum of tasks ranging from mechanistic numerical calculations to creative theory construction. Between these extremes lies a significant class of "algorithmic computations": tasks that are too complex for a single deterministic computer program to solve generically due to problem-specific subtleties, yet not so creative that they require entirely new methods. Examples include perturbative calculations in quantum field theory and identifying the degrees of freedom (dof) in modified gravity theories. While Computer Algebra Systems (CAS) assist in these tasks, they remain time-consuming and prone to human error in setup and interpretation. This work investigates whether frontier Large Language Models (LLMs), equipped with a CAS runtime and informed by in-context learning, can reliably automate these algorithmic tasks. Specifically, the authors focus on identifying the propagating degrees of freedom for cosmological perturbations in various modified gravity theories.

Methodology
The authors developed a framework interfacing the frontier LLM Claude Opus 4-6 with the CAS Maple. The system operates in a read-eval-print loop (REPL) where the LLM generates Maple commands based on the problem statement and context, executes them, and iterates until a solution is found or a limit is reached.

• Test Suite: The evaluation utilized nine distinct problems involving scalar-tensor and tensor theories on both flat and cosmological backgrounds. These problems were designed to be research-grade, featuring subtleties such as higher-derivative terms, branching solutions for background equations, and constraints that require careful handling (e.g., the constrained scalar-field framework).
• In-Context Learning Strategies: The study tested four distinct context configurations to determine what optimizes performance:
  1. 10ex: A set of 10 detailed, step-by-step solved examples (60k tokens).
  2. 3broad: A reduced set of 3 representative examples (18k tokens).
  3. 3tailored: A modified set of 3 examples, specifically engineered to address common failure modes observed in initial trials (24k tokens).
  4. instruction: A general algorithmic description without code examples (2k tokens).
• Evaluation: Success was measured on a pass/fail basis across four stages: setup, background equation derivation, second-order action computation, and perturbation analysis. The study also tracked the number of interaction turns and Maple session restarts as indicators of efficiency and stability.

Key Results
The study demonstrates that a frontier LLM equipped with a CAS and appropriate context can competently solve complex algorithmic physics problems, performing at a level comparable to a first-year graduate student.

• Context Efficacy: Worked examples are essential for success. The "instruction-only" context resulted in the lowest success rate (3/9) and highest computational cost (average 67 turns), as the model struggled to translate abstract algorithms into executable code. The "10ex" context achieved a 5/9 success rate.
• Optimization via Tailoring: The "3tailored" context, which included examples specifically designed to counter observed failure modes (such as incorrect reduction of higher-order derivatives and misinterpretation of background spacetimes), yielded the highest success rate (7/9) and the most efficient execution (lowest average turns and restarts). Notably, this was the only configuration where the difficult problem sRi2Ft was solved correctly.
• Failure Modes: Common failures included:
  • Misinterpretation of Backgrounds: The model occasionally confused "flat background" with a flat Friedmann-Lemaître-Robertson-Walker (FLRW) metric rather than Minkowski space, or deemed trivial solutions "too simple" and unnecessarily switched to curved spacetime analysis.
  • Constraint Handling: Errors frequently occurred in resolving background equations of motion or reducing higher-derivative terms, often due to forgetting Lagrange multipliers.
  • Thinking Mode: Enabling the model's "thinking" mode (allowing 1024 thinking tokens) provided negligible improvement, as the model rarely utilized the budget to correct fundamental strategy errors.
• Model Behavior: The LLM exhibited strong goal-orientedness, often recognizing when a path was unproductive and restarting the CAS session. It also demonstrated an ability to handle branching solutions, often selecting the simpler branch to simplify analysis, though this sometimes led to missing valid physical branches.

Significance and Claims
The paper claims that algorithmic theoretical physics serves as a rigorous testing ground for long-context LLMs due to the need for "look-ahead" reasoning and the availability of obscure, research-grade test cases that are unlikely to be in training data.

The primary contribution is the establishment of practical guidelines for practitioners wishing to employ AI in theoretical physics. The authors conclude that:

1. Worked examples are superior to abstract instructions for enabling LLMs to perform routine but burdensome symbolic computations.
2. A small, targeted set of examples (tailored to specific failure modes) is more effective than a large, generic set of examples.
3. Human-in-the-loop collaboration remains necessary to catch specific interpretative errors (e.g., background metric definitions), but the LLM can autonomously handle the bulk of the symbolic derivation.

The authors modestly state that while they do not aim to replace human researchers, this approach offers a viable path to automating the "algorithmic" portion of theoretical research, potentially freeing physicists to focus on higher-level conceptual work. They suggest future work will explore Retrieval-Augmented Generation (RAG) setups to dynamically pull relevant examples into the context.