Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

Imagine a massive, complex machine (like a giant language model) that can write stories, solve math problems, or answer questions. Scientists have figured out how to find the specific "gears and levers" inside this machine that make it do a specific task. This is called localizing the circuit.

However, just finding the gears isn't enough. We still don't know exactly what each gear does or how they work together. Usually, a human expert has to spend hours staring at the machine, guessing what a part does, testing their guess, and trying again. This is slow, boring, and hard to scale.

This paper asks: Can we hire a robot detective (a Language Model Agent) to do this detective work for us?

Here is the breakdown of their experiment, using simple analogies:

1. The Problem: The "Black Box" Mystery

Think of the language model as a giant, locked safe. Scientists have already used special tools to find the specific tumblers (the "circuit") that open the safe. But they don't know what each tumbler does.

• The Old Way: A human detective tries to guess, "Maybe this tumbler turns the lock?" They test it. If it fails, they guess again. It takes a long time.
• The New Idea: Can we give the job to an AI detective that can think, write code to test its own theories, and fix its mistakes?

2. The Training Ground: A "Toy" Safe

To test if their AI detective works, the researchers couldn't just use a real, messy safe (a real-world language model) because they wouldn't know the "correct" answer to check against.

Instead, they built 84 "Toy Safes" (called AGENTICINTERPBENCH).

• These are small, artificial machines built from scratch.
• The researchers know exactly how they work because they built them from a simple set of instructions (like a recipe).
• They know exactly what every single gear is supposed to do. This allows them to grade the AI detective: "Did you guess right?"

3. The Detective: HYVE (The "Hypothesize, Validate, Explain" Loop)

The researchers created an AI agent named HYVE. HYVE doesn't just guess; it follows a strict detective routine for every single gear in the machine:

1. Observe: HYVE looks at the gear. "Hmm, when the input is 'x', this gear lights up bright red. When it's 'y', it stays dark."
2. Hypothesize: HYVE makes a guess. "I bet this gear is a 'Red Light Detector' that only turns on for 'x'."
3. Validate (The Test): This is the most important part. HYVE writes a computer program to test its guess. It might say, "Let's force this gear to stay dark even when the input is 'x' and see if the machine breaks."
   • If the machine breaks exactly as predicted, the guess is confirmed.
   • If the machine keeps working, the guess is wrong. HYVE has to go back to step 2 and try a new guess.
4. Explain: Once all gears are tested, HYVE writes a summary: "This machine is a 'Fraction Calculator' that counts how many 'x's appear in the sentence."

4. The Results: The Detective is Good, But Not Perfect

The researchers tested HYVE using four different "brains" (different large language models) to see which one made the best detective.

• Success: The AI agents were surprisingly good! They could correctly identify what the gears did about 79% of the time and describe the overall task about 83% of the time.
• The Bottleneck: The AI was great at making the initial guess (the hypothesis). The problem happened in the testing phase.
  • Sometimes the AI wrote a test plan that was too vague.
  • Sometimes the AI wrote code to run the test, but the code had bugs (like a detective writing a test plan but forgetting to bring the flashlight).
  • Analogy: It's like a detective who has a brilliant theory about who the killer is, but when they try to go to the crime scene to check the alibi, they get lost or lock themselves out of the building.

Who was the best?

• Claude-Sonnet was the best at actually running the tests without crashing (writing bug-free code).
• Gemini was the best at writing the final, easy-to-understand explanation.
• GPT-5.4 was great at planning the tests, but its code often failed to run.

5. The Real-World Test: The "Llama" Case Study

To see if this worked outside of their "Toy Safes," they tried HYVE on a real, naturally trained language model (Llama-3-8B) that does math.

• They gave the AI a circuit found by other humans that helps the model add three numbers together.
• Result: The AI detective successfully figured out that some gears were "transfer heads" (moving numbers around) and correctly identified that other gears were just "redundant" (they looked important but weren't actually needed).
• This proved the method works even on messy, real-world machines, not just the clean toy ones.

The Bottom Line

The paper concludes that AI agents are promising new tools for explaining how AI works. They can do the heavy lifting of guessing and testing. However, they aren't perfect yet. The main hurdle is reliability: the AI needs to get better at writing the code that actually tests its own theories without making mistakes.

Until the AI gets better at "doing the experiment" (the validation loop), human experts will still need to double-check the work. But this is a major step toward automating the understanding of complex AI brains.