Optimas: An Intelligent Analytics-Informed Generative AI Framework for Performance Optimization

1. Problem Statement

High-Performance Computing (HPC) performance optimization is currently a manual, time-intensive process requiring deep architectural expertise. While existing tools (profilers, analytical models) can identify performance bottlenecks (e.g., memory stalls, low occupancy), they stop short of generating actionable code changes.

• The Gap: Large Language Models (LLMs) show promise for code generation but struggle with performance optimization when provided only with source code. Without dynamic performance context (e.g., stall behavior, warp occupancy, memory throughput), LLMs produce syntactically valid but performance-neutral or detrimental code.
• The Challenge: Bridging the gap between raw, multi-gigabyte performance telemetry and LLMs' limited context windows, while ensuring generated code is semantically correct and specifically targets identified bottlenecks.

2. Methodology: The Optimas Framework

Optimas is a modular, fully automated, end-to-end generative AI framework built on a multi-agent workflow. It treats performance optimization as a data-driven reasoning problem, converting complex runtime profiles into validated code transformations.

Core Workflow Components:

1. Data Extraction & Insight Generation:

   • Kernel Selection: Automatically identifies the top 2–3 performance-critical kernels (accounting for >80% of runtime) to fit within LLM context limits.
   • Roofline Analysis: Classifies kernels as compute-bound or memory-bound by calculating arithmetic intensity and resource utilization ratios against hardware limits.
   • PC (Program Counter) Sampling: Localizes stalls to specific source lines and instruction types (e.g., stall_membar, stall_dependency). Optimas aggregates raw traces into concise summaries, retaining only dominant stalls (e.g., those contributing >30% of stalled cycles) to reduce data volume by ~99.9%.
   • Hardware Event Correlation: Uses a randomized greedy algorithm (based on Ensemble Orthogonal Matching Pursuit) to select the most informative hardware counters from thousands of available metrics, mapping them to natural language descriptions.

2. Prompt Construction:

   • Combines the annotated kernel code with the structured diagnostic summaries (Roofline, PC Sampling, Hardware Counters).
   • Guardrails: The prompt explicitly restricts edits to relevant code regions, prohibits signature changes, and requires the model to cite specific diagnostic evidence for every optimization.
   • Backends: Supports commercial APIs (GPT-5, Gemini) and local models (Llama 3.1).

3. Evidence-Aligned Reasoning (EAR):

   • A closed-loop system where an Evaluation Agent compiles the generated code, executes it, and validates bit-for-bit correctness against reference outputs.
   • Feedback Loop: If compilation fails, the agent retries up to three times using error logs; persistent failures are flagged for manual review.

Novel Metrics:

The paper introduces Evidence-Aligned Reasoning (EAR) metrics to evaluate the quality of LLM reasoning:

• Evidence Coverage: Measures if the model explicitly references diagnostic clues in its justification.
• Localization Agreement: Assesses if code edits occur at the specific lines identified as hotspots by profiling.
• Directional Consistency: Verifies if the performance changes (e.g., reduced stall counters) align with the direction implied by the diagnostics.

3. Key Contributions

1. Multimodal Data Mining Pipeline: A method to distill complex HPC logs (Roofline, PC Sampling, Hardware Counters) into concise, actionable insights for LLMs.
2. Evidence-Aligned Reasoning (EAR) Metrics: A novel framework to quantify the consistency between diagnostic signals and generated code edits, moving beyond simple speedup metrics.
3. Large-Scale Evaluation: A comprehensive study involving 3,410 experiments across 10 benchmarks and two HPC proxy applications (XSBench, SW4Lite) on NVIDIA GPUs.
4. Open-Source Corpus: A curated dataset aligning performance diagnostics, code transformations, and measured outcomes to support future research in retrieval-augmented optimization.
5. The Optimas Framework: A backend-agnostic, closed-loop system that democratizes access to performance engineering.

4. Results

The framework was evaluated on NVIDIA H100 GPUs using GPT-5, Gemini-2.5-pro, and Llama 3.1:70B.

• Performance Gains: Optimas achieved performance improvements in 98.82% of experiments, with average gains ranging from 8.02% to 79.09%.
• Correctness: The framework generated 100% correct code (compilable and functionally valid) across all successful experiments.
• Model Comparison:
  • GPT-5: Achieved the highest evidence coverage (83.1%) and largest speedups, particularly on complex multi-kernel applications. However, it occasionally "over-optimized" by applying additive transformations beyond saturation.
  • Llama 3.1:70B: Showed lower evidence coverage (49.2%) but produced stable, conservative gains with zero hallucinations, proving that open-source models can be effective under strict guardrails.
  • Gemini-2.5-pro: Performed comparably on large kernels but struggled with fine-grained stall-sensitive kernels due to context dilution.
• Diagnostic Synergy: Combining all three diagnostic sources (PC + IA + Roofline) yielded the best results for complex applications with mixed bottlenecks. However, the benefit was not strictly monotonic; safety guardrails prevented unsafe rewrites when sources conflicted.
• Generalization: Optimizations generated from a default configuration generalized successfully across 68 different input configurations for most applications (e.g., Accuracy kernel showed stable 43–45% gains across all inputs).
• Overhead: The optimization process is highly efficient. While profiling takes hours (e.g., 16–24 hours for SW4Lite), the Optimas iteration (summarization, inference, compilation, validation) takes <10 minutes, representing <1.5% of the data collection time.

5. Significance

• Democratization of HPC: Optimas transforms performance optimization from an expert-only task into an automated workflow, allowing developers without deep architectural knowledge to achieve significant speedups.
• Closing the Loop: It is the first framework to integrate diverse analytic sources directly into an automated code optimization engine, effectively bridging the gap between diagnosis and resolution.
• Educational Value: By requiring models to cite diagnostic evidence, the system serves as an educational tool, making the causal links between hardware diagnostics and code transformations explicit.
• Scalability: The system successfully handles real-world, complex HPC applications (like seismic wave propagation and Monte Carlo neutron transport), proving its viability beyond simple microbenchmarks.

In conclusion, Optimas demonstrates that when LLMs are guided by rigorous, multimodal runtime diagnostics, they can reliably generate high-performance code transformations, significantly reducing the manual effort required for HPC optimization.