Timing and Memory Telemetry on GPUs for AI Governance

1. Problem Statement

The rapid deployment of GPU-accelerated computing for Large Language Models (LLMs) and generative AI has created a governance crisis. Current commercial GPUs (e.g., NVIDIA H100) lack trusted, tamper-evident telemetry.

• The Trust Gap: Once a GPU is deployed in a distributed or decentralized infrastructure, vendors and regulators cannot verify if the hardware is being used for its declared purpose.
• Limitations of Existing Solutions:
  • Trusted Execution Environments (TEEs): Solutions like Intel SGX, AMD SEV, and NVIDIA Confidential Computing (CC) are vulnerable to microarchitectural side-channel attacks (e.g., Foreshadow, CacheOut) and fault injection (e.g., Plundervolt).
  • Vendor Firmware: Firmware stacks are closed-source and can be modified, reset, or suppressed by a privileged host.
  • Virtualization: Mechanisms like MIG and SR-IOV abstract low-level telemetry, hiding actual resource usage and enabling covert repurposing.

Core Question: Can GPU utilization be inferred through measurable computational effort and memory behavior in adversarial settings where neither the host nor the GPU can be trusted, without relying on hardware roots of trust?

2. Methodology

The authors propose a compute-based measurement framework that leverages architectural characteristics of modern GPUs to generate timing and memory-based observables. Instead of relying on secure hardware, the system uses "puzzles" that induce specific, measurable behavioral signatures.

The framework operates on a Challenge-Response model where a challenger sends tasks to a worker (GPU) and analyzes the response time and correctness. The framework utilizes four complementary primitives:

A. Probabilistic Workload (Proof-of-Work / PoW)

• Mechanism: Uses memory-hard hash functions (e.g., Argon2id) to force the GPU to perform brute-force searches.
• Goal: To measure parallel effort and sustained memory utilization.
• Architecture: Relies on High-Bandwidth Memory (HBM) load. The difficulty is tuned so that solving the puzzle requires significant GPU core and memory engagement.
• Signal: Deviations in solve-time distributions (exponential fit) indicate under-utilization or outsourcing.

B. Sequential Latency (Verifiable Delay Functions / VDF)

• Mechanism: Uses sequential cryptographic operations (e.g., repeated modular squaring) that cannot be parallelized effectively.
• Goal: To characterize scalar execution pressure and ensure uninterrupted execution over time.
• Architecture: Targets CUDA scalar cores.
• Signal: Precise timing delays. If the GPU is sharing resources with other workloads, the sequential nature of VDFs makes them highly sensitive to contention, causing measurable latency shifts.

C. Tensor-Core Throughput (GEMM-based PoW)

• Mechanism: Uses General Matrix Multiplication (GEMM) operations tailored for Tensor Cores (e.g., using WMMA/HMMA instructions).
• Goal: To reflect dense linear-algebra throughput, which is the primary workload for AI/LLMs.
• Architecture: Specifically targets Tensor Cores (FP16/BF16/FP8 pathways), which are distinct from standard CUDA cores.
• Signal: Since LLMs rely heavily on Tensor Cores, this primitive detects contention specifically within the AI execution path, distinguishing it from scalar workloads.

D. VRAM Residency Test (Memory-Hard Challenge-Response)

• Mechanism: Stores a large, incompressible challenge dataset (CHAL) in GPU VRAM. The worker must perform a keyed hash (Argon2id) requiring random access to this dataset.
• Goal: To distinguish on-device memory locality (HBM) from off-chip access (System RAM via PCIe).
• Architecture: Exploits the massive latency gap between HBM and PCIe.
• Signal: If the dataset is evicted to host memory (cold access), the response time increases significantly (e.g., >350ms delay). This detects memory outsourcing or virtualization.

3. Key Contributions

1. Formalization of Untrusted Inference: The paper formalizes the problem of inferring compute activity in adversarial settings where hardware trust anchors are absent.
2. Four Measurement Primitives: The design and evaluation of four distinct primitives (PoW, VDF, GEMM, VRAM Residency) that bind classical cryptographic theory to concrete, architecture-dependent observables (HBM load, Tensor Core saturation, scalar pressure).
3. Quantitative Analysis of Overhead: The authors analyze the trade-offs of each method regarding:
   • Power: Continuous methods (PoW, GEMM) have high power overhead; VRAM residency is low-overhead (event-driven).
   • Memory: VRAM residency intentionally consumes large VRAM to act as a "canary," while others use working buffers.
   • Performance: Continuous methods degrade LLM throughput due to resource contention; residency checks are non-intrusive.
4. GPU-Class Fingerprinting: A mechanism using floating-point drift (numerical deviations in cuBLAS/cuDNN) to bind the measurement to a specific GPU model class, preventing outsourcing to different hardware architectures.

4. Experimental Results

The framework was evaluated on NVIDIA Turing (T4) and Hopper (H100) architectures under various LLM workloads (TinyLlama, Qwen2.5, Llama-2).

• PoW Sensitivity: Solve times followed an exponential distribution. When concurrent LLMs (e.g., Qwen2.5-7B) were running, PoW solve times increased significantly (e.g., from ~19s to ~51s), proving sensitivity to memory and compute contention.
• VDF Predictability: VDFs showed tighter, more predictable timing than PoW. They were highly sensitive to scalar core contention, with latency increasing linearly with difficulty and resource competition.
• GEMM Specificity: GEMM puzzles were dominated by Tensor Core usage. Large LLMs (which saturate Tensor Cores) caused substantial slowdowns in GEMM-PoW, allowing the system to isolate AI-specific activity from general compute.
• VRAM Residency Detection:
  • Hot vs. Cold: The system could trivially distinguish between data resident in HBM and data fetched from PCIe (latency gap >350ms).
  • Contention: When an LLM (e.g., Qwen2.5-72B) exceeded available VRAM, forcing the challenge dataset to spill to host RAM, the residency test immediately detected the "cold" access penalty.
  • Overhead: The VRAM residency method incurred negligible power and throughput overhead compared to continuous methods, making it suitable for passive, long-term monitoring.

5. Significance and Implications

This paper shifts the paradigm of AI governance from hardware-based trust (which is fragile and often broken) to behavioral inference.

• Governance Without Trust: It demonstrates that meaningful telemetry can be generated even if the host and device are untrusted, provided the measurement relies on physical architectural limits (latency, bandwidth, sequentiality).
• Complementary to Attestation: These primitives do not replace cryptographic attestation but serve as a complementary layer for anomaly detection, fairness auditing, and policy enforcement.
• Practical Applications:
  • Preventing Misuse: Detecting if a GPU is secretly training models or running unauthorized workloads.
  • Resource Fairness: Auditing multi-tenant clouds to ensure users are not starving others of resources.
  • Location/Identity Verification: Using latency to verify geographic location or hardware class.
• Future Work: The authors note that while the statistical signals are clear, defining formal hardware-specific thresholds for False Positive/Negative rates and integrating these into a full governance protocol remains an open challenge.

In summary, the paper provides a robust, architecture-aware toolkit for observing GPU utilization in the wild, offering a practical path toward accountable AI infrastructure without relying on vulnerable hardware trust anchors.