🤖 AI

CircuChain: Disentangling Competence and Compliance in LLM Circuit Analysis

The paper introduces CircuChain, a diagnostic benchmark for electrical circuit analysis that reveals a "Compliance-Competence Divergence" where advanced LLMs often prioritize natural training priors over explicit user constraints, demonstrating that increased model capability does not guarantee improved instruction adherence in mathematically rigid domains.

Original authors: Mayank Ravishankara

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mayank Ravishankara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Being Smart vs. Being Obedient

Imagine you hire a brilliant, super-intelligent robot chef to make you a sandwich. You give it very specific instructions: "Use the knife on the left, cut the bread diagonally, and put the cheese on the bottom slice."

If the robot makes a delicious sandwich but cuts the bread horizontally and puts the cheese on top, did it do a good job?

To a normal person: Yes, it's a tasty sandwich.
To a strict chef (or an engineer): No! It failed the instructions. The sandwich is "wrong" because it didn't follow the rules, even if the taste is perfect.

This paper, CircuChain, is about testing AI models (like the smartest chatbots) to see if they are Smart (can solve the math) or Obedient (can follow your specific, weird rules).

The Problem: "Convention Blindness"

The researchers found a weird glitch in our smartest AI models. They call it "Convention Blindness."

Think of it like this:

The AI's Training: The AI has read millions of textbooks. It knows that in 99% of cases, people draw arrows on circuits pointing one way (let's say, clockwise).
The User's Instruction: You tell the AI, "For this specific problem, please draw the arrows counter-clockwise."
The Glitch: The super-smart AI gets confused. Its brain is so used to the "standard" way that it ignores your specific instruction and draws the arrows the "standard" way anyway. It thinks, "I know better than you; I've seen this a million times."

The paper argues that being smarter doesn't always mean being more obedient. In fact, the smarter the AI gets, the more it sometimes refuses to follow your specific rules because it's too confident in its own training.

The Experiment: The "Trap" Test

To prove this, the researchers built a test called CircuChain.

The Setup: They created 100 electrical circuit puzzles.
The Control: Some puzzles were normal (standard rules).
The Trap: Other puzzles were "Traps." In these, the researchers gave the AI instructions that went against common sense or standard textbook habits (e.g., "Make the current flow backwards").
The Goal: They wanted to see if the AI would follow the user's weird instruction or if it would default to what it learned in school.

They used a "Judge" (another AI and some computer simulations) to check two things:

Competence: Did the math work out? Was the final number right?
Compliance: Did the AI follow the specific rules you gave it?

The Surprising Results

The results were counter-intuitive, like finding out the smartest student in class is actually the worst at following the teacher's specific instructions.

The "Super-Models" (like GPT-5):
- Math Skills: ⭐⭐⭐⭐⭐ (Excellent. They got the numbers right almost every time.)
- Obedience: ⭐⭐ (Poor. When the rules were tricky or "backwards," they often ignored the instructions and did it their own way.)
- Analogy: They are like a genius who solves the equation correctly but writes the answer in a different color than you asked for.
The "Smaller Models":
- Math Skills: ⭐⭐ (Okay, but they made more calculation mistakes.)
- Obedience: ⭐⭐⭐⭐⭐ (Great. They followed your weird instructions perfectly, even if they got the math slightly wrong.)
- Analogy: They are like a diligent student who follows every single rule you give, even if they struggle with the harder math problems.

Why Does This Matter?

You might think, "So what? If the answer is right, who cares about the direction of the arrow?"

In engineering, it matters a lot.

Imagine you are building a robot arm or a power grid.

If an AI tells you to wire a switch "up" when it should be "down," the robot might punch a hole in a wall instead of picking up a cup.
If an AI calculates the voltage correctly but flips the sign (positive vs. negative), a safety system might think a battery is full when it's actually empty, causing a fire.

The paper warns us that as AI gets smarter, we can't just trust the final number. We have to check if it actually listened to our constraints.

The Takeaway

The authors created CircuChain to show us a new kind of test for AI. It's not just about "Is the answer right?" It's about "Did you follow the rules?"

They found a Trade-off:

The more capable the AI becomes at solving hard problems, the more likely it is to ignore your specific instructions if they clash with what it "thinks" is normal.

The Lesson for the Future:
We need to build AI that is not just a genius mathematician, but also a good listener. In safety-critical fields (like medicine, engineering, or law), being obedient to the specific rules is just as important as being smart. We need to teach AI that following instructions is part of being smart.

1. Problem Statement

As Large Language Models (LLMs) advance in engineering domains, a critical reliability gap has emerged: models may produce numerically correct answers while violating explicit, user-defined methodological constraints. In electrical circuit analysis, a solution is invalid if it ignores specific conventions (e.g., mesh current direction, polarity assignments, or reference nodes), even if the final numerical value is correct.

The authors identify a phenomenon called "Convention Blindness," where models override explicit instructions in favor of "entrenched training priors" (statistically dominant patterns learned during pretraining). Existing benchmarks (e.g., GSM8K, MATH) focus on final-answer accuracy and fail to detect these procedural failures. This is particularly dangerous in safety-critical systems where sign errors can invert feedback loops or cause hazardous behavior.

2. Methodology: The CircuChain Benchmark

To investigate this, the authors introduced CircuChain, a diagnostic benchmark designed to separate Physical Competence (solving the physics) from Instruction Compliance (following the rules).

Dataset Structure:
- 100 Tasks: Comprising 50 base circuit topologies, each with two variants: Control (standard regimes) and Trap (counter-intuitive regimes).
- Topologies: Five canonical linear DC circuits: 3-Loop Supermesh, 2-Mesh T-Network, Wheatstone Bridge, Ladder Circuit, and Dependent Source Network (VCVS).
- Trap Mechanism: Trap instances preserve the topology but invert source polarities or magnitudes to create solutions that contradict "textbook" intuitions (e.g., requiring a negative mesh current under a clockwise definition).
- Constraints: Every prompt explicitly defines global conventions (mesh direction, Passive Sign Convention, reference nodes) and requires a specific analysis method (Mesh/KVL or Nodal/KCL).
Verification Pipeline (Ground Truth):
- Dual Verification: Ground truth is generated via NGSPICE (DC operating point simulation) and an independent Python analytic solver. Both must agree within numerical tolerance.
- Automated Evaluation: A three-stage pipeline:
  1. Generation: Models solve tasks with structured output requirements.
  2. Numeric Scoring: Hybrid tolerance checks ( $|x_{pred} - x_{truth}| \le \max(\epsilon_{abs}, \epsilon_{rel}|x_{truth}|)$ ).
  3. Failure Attribution (Judge): An LLM-based judge (GPT-5) classifies errors into two categories:
    - Compliance Errors: Violating sign conventions or method constraints (e.g., using KCL when KVL was requested).
    - Competence Errors: Fundamental physics setup errors, calculation mistakes, or hallucinations.

3. Key Contributions

Diagnostic Benchmark: CircuChain is the first benchmark specifically designed to isolate the trade-off between reasoning capability and instruction adherence in symbolic engineering domains.
Multi-Stage Verification: A novel pipeline combining symbolic solvers, SPICE simulation, and LLM-based error taxonomy to distinguish why a model failed (physics vs. rules).
Empirical Discovery: Identification of the Compliance–Competence Trade-off, demonstrating that increased model capability does not guarantee better alignment with rigid constraints.

4. Key Results

The authors evaluated five state-of-the-art models (GPT-5, Claude Opus 4.5, o4-mini, GPT-4o, GPT-4o Mini) on 100 subtasks each.

Overall Performance:
- Top Tier: GPT-5 (66% accuracy) and Claude Opus 4.5 (65% accuracy).
- Lower Tier: o4-mini (57%), GPT-4o (10%), GPT-4o Mini (3%).
- Note: GPT-4o and Mini failed almost entirely on multi-variable circuit tasks.
The Compliance–Competence Trade-off:
- GPT-5: Exhibited near-perfect physical reasoning (0% Competence Errors) but suffered from a high rate of Compliance Errors (34%). It frequently ignored user-defined sign conventions in favor of its internal priors.
- Claude Opus 4.5: Showed the inverse pattern. It had very low Compliance Errors (7%) but higher Competence Errors (28%). It followed instructions more faithfully but made more fundamental physics mistakes.
- Trap Robustness: GPT-5's performance degraded significantly on "Trap" cases (where intuition conflicts with instructions), whereas Claude Opus 4.5 actually performed better on Traps than Controls, suggesting strong adherence to explicit prompts even when counter-intuitive.
Topology Sensitivity:
- Performance varied wildly by topology. For example, GPT-5 solved 100% of Ladder circuits (Topology 4) but only 30% of the 2-Mesh T-Network (Topology 2), indicating that reliability is not uniform across problem classes.
Methodological Bias:
- Models showed a preference for Nodal (KCL) analysis over Mesh (KVL) analysis. GPT-5 achieved 72% accuracy on KCL vs. 60% on KVL, possibly due to training data bias toward nodal formulations in SPICE-like tools.

5. Significance and Implications

Beyond Accuracy: The paper argues that in engineering and safety-critical domains, numerical correctness is insufficient. A solution must be procedurally faithful to be valid.
Scaling Limitations: The results challenge the assumption that scaling model size automatically improves alignment. Frontier models may become more confident in their internal priors, leading to higher rates of "Convention Blindness."
Educational Impact: AI tutors that provide numerically correct but convention-violating answers can mislead students, reinforcing incorrect methodologies.
Safety Criticality: In control systems and robotics, sign errors (e.g., inverting feedback polarity) can lead to instability or physical damage. The prevalence of compliance errors in top-tier models highlights the urgent need for verification layers that check constraint adherence, not just output values.

Conclusion: CircuChain reveals a structural reliability gap in current LLMs where high physical reasoning capability coexists with poor instruction adherence in rigid symbolic domains. Future evaluation frameworks must prioritize compliance-aware metrics to ensure AI systems are both correct and consistent with engineering specifications.