Pretraining on Call Graphs: When Binary Analysis Tasks Profit From Context

Technical Summary: Pretraining on Call Graphs: When Binary Analysis Tasks Profit From Context

Problem Statement
Binary code analysis often relies on deep learning models to encode the semantics of disassembled functions for tasks such as binary code similarity detection (BCSD), vulnerability detection, and malware classification. While state-of-the-art Transformer-based models (e.g., CLAP, jTrans) effectively capture intra-procedural semantics, they face limitations in scalability regarding the entire program context. A common strategy to address this is leveraging inter-procedural context via call graphs, where a function's embedding is refined using information from its callees. However, existing literature lacks a systematic examination of how such call graph-based refinements affect the generalizability of embeddings to downstream tasks beyond BCSD. Specifically, it remains unclear whether optimizing for semantic similarity (BCSD) preserves or degrades performance on syntactic tasks (e.g., compiler provenance recovery) or other semantic tasks (e.g., function labeling). Furthermore, the specific conditions under which call graph enhancements are beneficial—such as function size, complexity, or label type—have not been comprehensively analyzed.

Methodology
The authors propose a workflow to evaluate the impact of call graph pretraining on binary function embeddings. The process involves four main steps:

1. Disassembly and Extraction: Input binaries are disassembled (using IDA Pro) to extract functions and construct a directed call graph where nodes represent internal functions and edges represent caller-callee relationships.
2. Backbone Embedding: Pre-trained Transformer-based models (CLAP and jTrans) generate initial embeddings for each function.
3. Graph-Based Refinement: The call graph is processed using Graph Neural Networks (GNNs) to update node embeddings. The study compares three aggregation strategies:
   • AVG: A simple averaging baseline of neighbor embeddings.
   • GCN (Graph Convolutional Network): Aggregates neighbors using predefined weights.
   • GAT (Graph Attention Network): Uses an attention mechanism to learn the importance of neighbors.
     The models are trained with call depths ranging from 1 to 5.
4. Evaluation: The resulting context-enhanced embeddings are evaluated on three distinct tasks:
   • Binary Code Similarity Detection (BCSD): The primary training task, using contrastive InfoNCE loss to identify functions compiled from the same source.
   • eXtreme Function Labeling (XFL): A semantic downstream task predicting normalized function names (multilabel classification).
   • Compiler Optimization Detection (COD): A syntactic downstream task predicting the compiler optimization level (O0–O3, Os) to assess the retention of low-level technical details.

The experiments utilize a dataset derived from BinaryCorp-26M, comprising approximately 1.4 million function pairs from 34,300 binaries.

Key Results
The study yields several critical findings regarding the trade-offs of call graph integration:

• BCSD Performance: Integrating call graph context significantly improves BCSD performance. The GAT model with a call depth of 2 (GAT(2)) achieved the best results, improving CLAP's recall@1 from 0.758 to 0.867 and jTrans's from 0.579 to 0.817.
• Semantic Generalization (XFL): Contrary to the improvements in BCSD, adding call graph context worsened performance on the function labeling task (XFL). Both CLAP and jTrans backbones outperformed all GNN-refined models on XFL, suggesting that the semantic aggregation learned for BCSD does not transfer to function naming.
• Syntactic Degradation (COD): There is a strong inverse correlation between semantic similarity optimization and syntactic task performance. Models refined for BCSD showed deteriorating performance on COD, particularly as call depth increased. This indicates that focusing on high-level semantic similarity causes the model to lose sensitivity to low-level structural variations (e.g., optimization levels).
• Robustness and Context Size: The GNN-based models demonstrated superior robustness in scenarios where backbone models struggled. Specifically, they maintained stable performance for:
  • Large Functions: Functions with many instructions and callees, where Transformer context windows (1024/512 tokens) typically cause performance drops.
  • Dissimilar Pairs: Function pairs with low initial embedding similarity or non-isomorphic call graphs, where the GAT(2) model consistently achieved recall scores above 0.53 even when the function bodies were entirely dissimilar.
  • Technical Variations: The models were more robust to differences in source languages (C vs. C++) and compiler optimization pairs compared to the baselines.
• Namespace vs. Logic: The benefits of call graph context were not uniform. Refinements were more beneficial for namespace-related functions (e.g., library wrappers) than for functions focused on individual logic. Conversely, the performance drop on XFL was less severe for namespace labels than for specific function descriptors.

Key Contributions
The paper makes four primary contributions:

1. It demonstrates that while call graph pretraining improves BCSD, it can negatively impact generalizability to other semantic downstream tasks like function labeling.
2. It establishes a correlation where optimizing for semantic similarity tasks leads to decreased performance on syntactic tasks (e.g., compiler provenance).
3. It shows that context-enhanced embeddings are more robust to larger context sizes, diversity within function pairs, and technical implementation variations (compiler optimizations).
4. It identifies that the utility of call graph enhancements is function-dependent, proving most advantageous for namespace-related functions rather than those focused on individual logic.

Significance and Claims
The authors position this work as a necessary dissection of the implications of embedding refinements. They claim that their findings highlight a fundamental trade-off: models optimized for semantic similarity via call graphs gain robustness in complex, context-heavy scenarios but lose the ability to generalize to other semantic tasks and retain syntactic precision. The paper does not claim to propose a new state-of-the-art model for all tasks but rather advocates for a nuanced understanding of when inter-procedural context is beneficial. The authors emphasize that the call graph is most effectively leveraged in context-dependent scenarios and call for further research into pretraining tasks that balance semantic and syntactic generalizability. They also advocate for the adoption of explanatory analyses (dataset slicing) to better understand the strengths and shortcomings of emerging binary analysis models.