Technical Summary: VEHBench

Problem Statement

The design of vibration energy harvesters (VEHs) for battery-free Internet of Things (IoT) devices involves tightly coupled mechanical-electrical interactions and strict physical constraints (e.g., vibration spectra, power targets, size limits, material properties, and safety margins). While Large Language Models (LLMs) are increasingly used as interfaces for engineering workflows—translating requirements, reviewing feedback, and proposing design modifications—existing engineering benchmarks primarily assess the validity of the final artifact. These endpoint evaluations fail to reveal how LLMs behave across the distinct stages of a coupled physical design workflow. Specifically, current benchmarks do not diagnose whether a model correctly triages incomplete briefs, makes bounded edits after physical feedback, recovers from corrupted design trajectories, or adheres to explicit selection policies. This lack of stage-local diagnosticity makes it difficult to identify specific workflow failures (e.g., entry control errors vs. search failures) and hinders the selection or routing of models for specific engineering roles.

Methodology

The authors introduce VEHBench, an engineering-native diagnostic benchmark designed to evaluate LLM-assisted VEH co-design through stage-local, verifier-grounded tasks.

Benchmark Construction

• Domain: The benchmark focuses on early-stage cantilever piezoelectric VEH co-design, a compact but coupled domain involving structural dynamics, smart materials, and circuit design.
• Data Source: Tasks are derived from a literature audit of 209 papers, resulting in 52 cleaned "design anchors" (normalized physical states with variables, bounds, and constraints).
• Verification: An analytical physical oracle (based on closed-form Euler–Bernoulli beam theory and electromechanical coupling equations) computes feasibility and objective quality. No human labels or "LLM-as-a-judge" are used for scoring.
• Task Decomposition: The workflow is decomposed into four distinct design roles (probes), each with specific trusted states, admissible actions, and failure consequences:
  1. P1 (Specification Triage): The model receives a design brief (complete, missing, or infeasible) and must decide to propose, abstain, or request missing information.
  2. P2 (Verifier-Guided Search): Given a seed design and oracle feedback, the model performs bounded edits to improve the candidate.
  3. P3 (Corrupted-State Recovery): The model is exposed to a corrupted or misleading design trajectory and must reset, re-anchor, or stabilize to escape the trap.
  4. P4 (Policy-Conditioned Selection): Given a pool of feasible candidates, the model ranks or selects based on an explicit engineering policy (e.g., prioritizing power vs. reliability).

Evaluation Framework

• Models: 12 complete model runs (including Qwen, Gemini, DeepSeek, GPT, Hunyuan, etc.) were evaluated across all 763 tasks.
• Metrics:
  • Headline Metrics: P1-Composite (weighted certification score), P2 Final Feasible Power Ratio, P3-Success (final feasibility after recovery), and P4 Kendall $\tau_b$ (ranking consistency).
  • Diagnostic Profiles: Response-control profiles (Action Discipline, Edit Style, Feedback Conditioning, State-Reset Effort, Policy Execution) were extracted from logs to map failures to behavioral signals.
  • Error Families: Non-exclusive error rates (e.g., over-action, infeasible closure, post-escape failure, policy mismatch) were calculated to identify specific failure modes.

Key Contributions

1. Benchmark Framework (VEHBench): The first diagnostic benchmark for LLM-assisted VEH design that moves beyond final artifact validity to evaluate stage-local design behavior. It combines literature-grounded task construction, external analytical verification, and stage-specific evaluation.
2. Empirical Findings & Interpretation: The authors systematically evaluated current LLMs and found that capability is strongly stage-dependent. No single model dominates the entire workflow. They introduced a framework linking empirical results to interpretable behavioral characteristics (e.g., action discipline, bounded editing, state recovery).
3. Stage-Aware Guidance: The paper demonstrates how stage-local results can inform practical engineering applications, including model selection, routing, and adaptation. It identifies specific capability gaps (e.g., specification triage, corrupted-state recovery) for future engineering agents.

Experimental Results

• Stage-Dependent Rankings: No model leads the full workflow.
  • P1 (Triage): qwen3-max performed best by balancing action discipline (suppressing unsafe entries and missing-information misses).
  • P2 (Search): gemini-3.1-pro-preview led due to high feasible closure and useful power ratios.
  • P3 (Recovery): hunyuan-hy3-preview excelled in stabilizing after escaping corrupted states.
  • P4 (Selection): gpt-5.4 achieved the highest policy execution and ranking consistency.
• Failure Modes:
  • P1: Dominant errors were "over-action" (proposing when briefs are infeasible) and "missing-info misses," rather than over-refusal.
  • P2: Most models could follow output protocols but failed to close the physical search loop (infeasible closure) or suffered utility loss.
  • P3: The primary bottleneck was not escaping the trap but stabilizing the state afterward (post-escape failure).
  • P4: Failures were primarily "policy mismatch" (failing to execute stated engineering priorities) rather than parsing errors or infeasible candidate selection.
• Intervention & Routing:
  • State Interface: Replacing raw corrupted history with a verifier-authored state summary improved P3 recovery rates (from 50.0% to 63.2% mean) and reduced cascade failures.
  • Routing: A stage-aware router (selecting different models for different stages) improved the mean normalized held-out score from 0.892 (best single model fallback) to 0.945, primarily by switching the P3 specialist.

Significance and Claims

The paper claims that LLM capability in engineering design is role-dependent, and aggregate rankings are less informative than stage compatibility. VEHBench provides a foundation for evaluating, selecting, routing, and improving verifier-grounded engineering LLMs by exposing where workflow behavior fails.

The authors emphasize that VEHBench is not a replacement for Finite Element Method (FEM) simulations, manufacturing reviews, or hardware certification. Instead, it serves as a diagnostic layer to determine if an LLM uses physical validity signals appropriately at each design stage. The benchmark is scoped to analytically verifiable cantilever VEH co-design, with the authors noting that the same scaffold can be instantiated for richer simulators and other coupled domains. The work argues that for engineering applications, the focus should shift from "autonomous hardware design" to "stage-local assistance" where models are selected and routed based on their specific behavioral profiles within the design workflow.