IndustryCode: A Benchmark for Industry Code Generation

1. Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable proficiency in general code generation (e.g., web development, algorithmic puzzles), existing benchmarks (such as HumanEval, MBPP, or SWE-bench) fail to evaluate their capabilities in real-world industrial scenarios.

• Limitations of Current Benchmarks: They are often confined to single domains (e.g., general software), single programming languages, and lack the rigorous constraints of vertical industries (finance, aerospace, automation).
• The Gap: Industrial coding requires handling domain-specific physical constraints, specialized languages (MATLAB, Stata), high numerical precision, and complex system decomposition. Current benchmarks do not test an LLM's ability to generalize across these diverse, high-stakes environments or to orchestrate multi-step engineering workflows.

2. Methodology: The IndustryCode Benchmark

The authors introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages.

A. Dataset Construction

• Scale: The dataset contains 125 Main Problems (complex industrial challenges) hierarchically decomposed into 579 Sub-problems.
• Source: Data is derived from authentic, proprietary production code used by industry practitioners. To prevent data contamination (memorization), the authors manually reconstructed the code, injected adversarial complexity, and verified it against strict functional tests.
• Domains: Covers four major sectors: Finance, Automation, Aerospace, and Remote Sensing, spanning 20 sub-fields (e.g., Chemical Engineering, Biomedical, Mining).
• Languages: Supports four distinct programming languages reflecting industry stacks: Python, C++, MATLAB, and Stata.

B. Hierarchical Problem Structure

The benchmark mimics real-world engineering workflows through a two-level structure:

1. Main Problem: Represents a complete engineering project with global objectives, inputs, and outputs.
2. Sub-problems: The Main Problem is broken down into modular, functionally independent tasks.
   • Evaluation Strategy: Models must implement code for sub-problems and synthesize them into a unified solution. This tests both granular coding accuracy and high-level system orchestration.

C. Annotation and Quality Control

A rigorous Human-AI collaborative pipeline was used:

• Difficulty Enhancement: Problems were modified to include complex mathematical logic, non-linear optimization, and strict engineering constraints (e.g., numerical stability, specific design patterns).
• Evaluation System:
  • Execution-based Numerical Validation: Dynamic testing using input-output pairs to verify numerical precision.
  • LLM-based Semantic Evaluation (LLM-Judge): A specialized judge model evaluates functional equivalence and adherence to industrial coding standards (modularity, control flow).
• Iterative Verification: A three-stage process involving automated testing, domain expert review, and closed-loop refinement to ensure all 579 sub-problems yield results within engineering tolerance.

3. Key Contributions

1. First Cross-Domain Industrial Benchmark: IndustryCode bridges the gap between general code generation and specialized industrial applications, covering diverse sectors and languages.
2. Hierarchical Evaluation Framework: It introduces a novel "Main Problem vs. Sub-problem" evaluation metric to distinguish between atomic coding skills and complex system integration capabilities.
3. High-Fidelity Data: By sourcing from real production code and applying adversarial hardening, the benchmark minimizes data contamination risks and reflects genuine engineering challenges.
4. Comprehensive Error Analysis: The paper provides a granular taxonomy of failure modes specific to industrial code, such as "Cognitive Over-correction" and "Contextual Interference."

4. Experimental Results

The authors evaluated 20 state-of-the-art LLMs (including Claude 4.5, GPT-5, Qwen3, and DeepSeek) on IndustryCode.

A. Overall Performance

• Top Performer: Claude 4.5 Opus achieved the highest accuracy: 68.1% on Sub-problems and 42.5% on Main Problems.
• Performance Gap: There is a significant "Execution Gap." Even the best models show a ~25% drop in accuracy when moving from atomic sub-tasks to full system integration (Main Problems), indicating struggles with long-horizon planning and error recovery.
• Open vs. Closed Models: While closed-source models generally lead, open-weight models like Qwen3-Max showed remarkable competitiveness, outperforming some proprietary models in specific domains (e.g., 70.4% in C++ sub-problems).

B. Domain and Language Analysis

• Domain Variance: Models perform best in mathematically intensive, algorithmic domains (Finance: 62.35%, IT: 59.18%) but struggle in highly specialized engineering fields requiring specific domain knowledge (Semiconductors: 33.50%, Naval Architecture: low accuracy).
• Language Disparity: Performance is heavily correlated with training data availability. Python and C++ show high accuracy, while Stata and MATLAB (less common in public training corpora) result in near-zero performance for many models.

C. Failure Mode Analysis

The study identified four primary failure mechanisms:

1. Syntax Errors (32.8%): The leading cause of failure, often due to strict industrial language constraints.
2. Problem Misinterpretation (30.2%): Failure to translate complex engineering requirements into logic.
3. Hallucination (19.6%): Fabricating non-existent APIs, particularly in domain-specific libraries.
4. Reasoning Failure (8.4%): Surprisingly rare, suggesting models have strong logical reasoning but lack specific vertical domain knowledge.

D. Impact of "Thinking" (Reasoning) Modes

• Trade-off: Activating "Thinking" modes (Chain-of-Thought) significantly reduced Reasoning Failures (from 11.8% to 0.8%) but caused a surge in Context Confusion (from 1.9% to 29.2%) and Syntax Errors.
• Mechanism: Extended reasoning often leads to "Cognitive Over-correction," where models apply generalized modern best practices that violate specific legacy industrial constraints (e.g., using unsigned integers where signed are required).

5. Significance and Conclusion

• Benchmarking Standard: IndustryCode establishes a gold standard for evaluating "Industrial AI," moving beyond toy problems to assess real-world utility in high-stakes sectors.
• Insight for Model Development: The results highlight that current LLMs are proficient at code generation but lack the domain-specific knowledge and state persistence required for complex industrial workflows.
• Future Directions: The paper suggests that future models need better handling of specialized languages, improved long-context state tracking (to avoid redefinition errors), and mechanisms to balance general reasoning with strict domain constraints.
• Availability: The dataset and evaluation code are made publicly available to foster further research in industrial code generation.

In summary, IndustryCode reveals that while LLMs are powerful tools, they are not yet "industrial-grade" assistants. Significant gaps remain in handling domain-specific constraints, multi-step system integration, and specialized programming languages.