SPARC: A Multi-Agent System for Electrical Circuit Question Answering

Technical Summary: SPARC – A Multi-Agent System for Electrical Circuit Question Answering

Problem Statement
Answering natural language questions about electrical circuit diagrams requires complex mathematical reasoning grounded in physics. While multimodal Large Language Models (LLMs) have achieved strong performance on general diagram-based question answering, they struggle significantly with electrical circuits, achieving only ~51% accuracy on existing benchmarks. The core difficulty lies in the fact that required equations are not provided; they must be derived from the circuit topology, component connectivity, and physical laws (e.g., Ohm's law, Kirchhoff's laws). Furthermore, real-world engineering demands not just accuracy but reliability; probabilistic LLM outputs offer no guarantee of correct reasoning, and verifying their mathematical validity typically requires the very domain expertise automation seeks to replace. Current approaches often fail because they attempt to solve these problems end-to-end or rely on generating symbolic code (e.g., Python/SymPy) that requires the model to explicitly formulate and solve governing equations, a process prone to error.

Methodology: The SPARC System
The authors present SPARC (SPICE Program Analysis and Reasoning for Circuits), a multi-agent system that grounds reasoning in executable, physics-based simulations using SPICE (Simulation Program with Integrated Circuit Emphasis), the de facto framework for circuit simulation. Rather than asking an LLM to solve the math directly, SPARC uses LLMs as an interface to translate questions and diagrams into SPICE programs, while the SPICE simulator (ngspice) performs the actual computation.

The system decomposes the task into three distinct stages, managed by specialized agents:

1. Simulation Setup:

   • A Planner Agent analyzes the natural language question and the circuit netlist to determine the number of required simulations. It decomposes complex queries (e.g., comparing two states or sweeping parameters) into independent sub-questions.
   • It constructs an initial state containing the circuit diagram, netlist, and a base SPICE program derived from the netlist.

2. Simulation Execution:

   • This stage involves generating executable SPICE programs for each sub-question. SPARC employs three specialized agents to modify disjoint sections of the base program:
     • Circuit Specification Agent: Updates component values (e.g., resistors, voltage sources) based on explicit values in the question or infers implicit values (e.g., "negligible ripple" implying large capacitance). It also handles missing device models.
     • Analysis Specification Agent: Selects the correct analysis type (DC, AC, or Transient) and configures parameters (e.g., frequency, time steps) based on domain-specific keywords in the question.
     • Output Specification Agent: Determines which measurements to extract (e.g., node voltages, currents) and formulates the appropriate SPICE directives (.print, .meas).
   • Error Handling: If a simulation fails, an Error Diagnoser Agent analyzes the simulator logs to identify the root cause (e.g., syntax errors, inconsistent parameters). It selectively re-invokes only the faulty agent to apply a targeted repair patch, rather than regenerating the entire program. This iterative loop continues until execution succeeds or a retry limit is reached.

3. Simulation Analysis:

   • Once all simulations execute successfully, an Answer Generation Agent aggregates the output logs. Using a Vision-Language Model (VLM) and a restricted set of domain-specific computation tools (e.g., for basic arithmetic or Ohm's law), it interprets the simulation results to derive the final answer to the original question.

Key Contributions

• SPARC Architecture: The introduction of a multi-agent system that synthesizes, executes, and analyzes SPICE programs, shifting the burden of mathematical reasoning from the LLM to a verified physics-based simulator.
• New Datasets: The creation of two datasets pairing circuit diagrams with complex QA pairs requiring mathematical analysis:
  • CktBench: An adaptation of EEE-Bench with 1,205 data points, augmented with corresponding netlists.
  • NetQ: A dataset of 495 QA pairs automatically generated from the AMSNet dataset, where questions are derived from simulation outputs to ensure ground-truth correctness.
• Structured Error Diagnosis: The implementation of a systematic error taxonomy that localizes failures to specific pipeline stages (setup, execution, analysis), enabling targeted debugging and interpretability.

Experimental Results
The authors evaluated SPARC against state-of-the-art baselines, including Chain-of-Thought (COT) prompting and code generation (CODE) approaches using SymPy, across various models (GPT-5.1, Claude Sonnet 4, GPT-4O, GLM-4.5V, Qwen3-VL).

• Accuracy: SPARC achieved 83% accuracy on CktBench and 81.4% on NetQ.
• Improvement: This represents an absolute improvement of 30.3% to 58.7% over zero-shot COT baselines and up to 60.8% over prior general-purpose visual QA methods.
• Model Agnosticism: The system provided the largest gains for weaker base models (e.g., up to 58.6% improvement for Qwen3 on NetQ), effectively narrowing the performance gap between weaker and stronger models.
• Executability: While direct prompting of LLMs to generate SPICE code resulted in non-executable programs in up to 99.9% of cases (executability as low as 0.07%), SPARC's multi-stage repair pipeline achieved over 95% executability.
• Error Analysis: The study found that the primary bottlenecks were not simulation execution but rather result interpretation (misinterpreting correct logs) and parameter specification (incorrectly mapping question constraints to circuit values).

Significance and Claims
The paper claims that SPARC demonstrates the effectiveness of simulator-native reasoning for complex circuit QA. By offloading mathematical reasoning to SPICE, the system achieves high accuracy and, crucially, reliability through verifiable computation. The authors emphasize that their approach addresses a unique and largely unsolved challenge in multimodal AI: bridging the gap between visual perception of circuit diagrams and rigorous physical reasoning. The work establishes that for domains requiring strict end-to-end correctness, decomposing the task into specialized agents that interact with domain-specific tools is superior to end-to-end LLM generation. The authors note that while netlist extraction remains a visual challenge, their system successfully addresses the underexplored problem of synthesizing executable simulation programs from natural language specifications.