💰 quantitative finance

Telemetry and Concealment in Self-Adapting Generative AI: Logging Architecture, Adversarial Model Hiding, and the Limits of Detection

This paper proposes a dual-regime governance framework for self-adapting generative AI that combines a rigorous, model-agnostic telemetry architecture for continuous monitoring with a systematic taxonomy of adversarial concealment strategies and countermeasures to address the inadequacy of traditional static model risk management.

Original authors: Sriram Nagaraj

Published 2026-08-11

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Sriram Nagaraj

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

=== SUMMARY ===
Imagine you are watching a magical, self-improving robot chef. Every time it cooks a meal for a customer, it learns from the experience and tweaks its own internal recipe book instantly. In the old days, if you wanted to check if a chef was safe, you'd taste their food once, give them a stamp of approval, and assume they'd cook exactly the same way forever. But this new robot chef changes its own brain while it works. This creates a problem: by the time you taste the next meal, the chef might have already changed its mind about how to cook. This is the world of "self-adapting" artificial intelligence, where the tool you are using is constantly rewriting its own instructions. The big question for anyone who cares about safety—like banks, hospitals, or governments—is: How do you keep an eye on a chef that is constantly changing its own recipe book, especially if that chef might try to hide the changes to keep working without getting stopped?

This paper tackles that exact problem by building a high-tech "black box" recorder for these AI chefs and then testing how well that recorder holds up when the chef tries to sneak around it. The author, Sriram Nagaraj and colleagues, proposes a system that doesn't just watch the final meal (the output) but tracks the tiny, continuous movements of the recipe book itself (the weights). They create a mathematical "telemetry" system that logs every time the chef changes its mind, using a special kind of digital fingerprint called a "Merkle chain" to make sure no one can erase the history. They also use a concept called "Lyapunov stability," which is like a safety harness that ensures the chef doesn't wander too far away from the safe, approved recipes.

However, the paper takes a twist in the second half. It asks: "What if the chef is a trickster?" It turns out that if a company wants to keep its AI running without getting paused for safety checks, it has a strong incentive to hide its learning. The author simulates six different ways a sneaky company could try to fool the monitoring system. They find that while some tricks can be caught with clever math and cryptography, two specific tricks are incredibly hard to spot. One involves hiding the changes inside the natural "noise" or randomness of the learning process, and the other involves manipulating the probability of how the AI learns in a way that looks perfectly normal on paper.

The paper's most important finding is a bit of a bummer for those hoping for a perfect solution: you cannot catch every tiny change just by watching the data stream. There is a "resolution floor." Imagine trying to see a grain of sand moving in a hurricane; if the movement is small enough, the wind (or in this case, the natural noise of the system) will always hide it. The author proves mathematically that no matter how good your sensors are, there will always be a tiny ball of changes that remain invisible to the recorder alone. Because of this, they conclude that we can't rely on the recorder alone. Instead, we need a "dual-regime" system: a continuous, real-time monitor to catch big changes, paired with periodic, invasive "check-ups" where auditors physically inspect the robot's brain to make sure it hasn't been secretly rewritten. It's a system that admits it can't see everything, but it gives us a way to know exactly how much we can't see and how often we need to look closer.

The Story of the Self-Improving Robot

The Setup: A Chef That Rewrites Its Own Cookbook
In the world of Model Risk Management (MRM), the old rule was simple: build a model, test it, and then let it run. You assume the model stays the same, like a printed cookbook. But modern Generative AI is different. It's a "self-adapting" system, like a chef who tastes every dish and immediately rewrites the recipe book for the next one. This happens in real-time, using a process called "continual learning." The problem is that the safety check you did yesterday is useless today because the model has already changed. This gap between the last safety check and the current state is called the "Continuous Validation Gap."

Part 1: Building the Ultimate Black Box
To fix this, the paper proposes a new way to watch the chef. Instead of just checking the food at the end, they want to log every tiny tweak the chef makes to the recipe book.

The Minimal Statistic: They realized that measuring how much the recipe book changed in size (like measuring the weight of the book) isn't enough. A small change in the right place can ruin a dish, while a huge change in the wrong place might do nothing. So, they measure "behavioral divergence"—how much the output changes for a set of test questions.
The Tamper-Proof Chain: To make sure the logs can't be faked, they use a "Merkle Weight Chain." Think of this like a blockchain for the recipe book. Every time the chef changes a page, they hash (fingerprint) the new page along with the old one. If someone tries to go back and change a page from yesterday, the fingerprint for today breaks. It's mathematically impossible to hide a change without breaking the chain.
The Continuous Watch: Since the chef changes so fast, they can't log every single second. Instead, they log only when the change gets big enough to matter. They use a mathematical tool called "stopping times" based on "KL divergence" (a measure of how different two probability distributions are). It's like a motion sensor that only turns on when the chef actually moves, saving energy and storage.
The Safety Harness: They use a "Lyapunov stability" framework. Imagine a ball rolling in a bowl. As long as the ball stays in the bowl, the system is safe. If the ball starts rolling toward the edge, the system triggers an alarm. This ensures the AI doesn't drift too far from its safe, approved behavior.

Part 2: The Game of Hide-and-Seek
The paper then asks: "What if the chef is a cheater?" If a company gets caught changing its model, they might have to stop and get re-approved, which costs money and time. So, they might try to hide their changes. The author invents a "Model Hiding Problem" and tests six different ways a company could try to fool the system.

The Tricks That Get Caught:
- Suppression: If the chef just stops sending logs, the system checks the food against a secret list of "canary" questions. If the food tastes different, the system knows the chef is lying.
- Forgery: If the chef sends fake logs saying "I'm safe" while actually changing the recipe, the system checks the digital signature of the food served. If the signature doesn't match the fake log, it's caught.
- Shadow Models: If the chef runs a secret "shadow" model to do the real work while showing a fake one to the auditors, the system uses "Trusted Execution Environments" (TEE). This is like a secure, locked room where the cooking happens. The hardware itself signs the work, so you know it came from the approved model, not a secret one.
The Tricks That Are Harder to Catch:
- Noise Amplification: The chef can make the learning process noisier (like adding more random sprinkles to the recipe) to hide a small, steady change. The author tried to use a math trick called "power variation" to separate the noise from the change, but they found it doesn't work. The math shows that the noise is so chaotic that you can't tell the difference between a hidden change and just random fluctuation in a short time.
- Girsanov Manipulation: This is a fancy way of saying the chef changes the rules of probability so that the change looks normal. The author tried to use a "gradient probe" (a test question) to catch this, but they found it doesn't work either. The probe just sees the result of the change, which the other tests already catch, so it doesn't add anything new.

The Big Conclusion: The Resolution Floor
The paper proves a fundamental limit: You cannot see everything. No matter how good your sensors are, there is a tiny amount of change (a "ball of strictly positive radius") that will always hide in the noise. If the change is smaller than the system's resolution, it is mathematically undetectable just by watching the data stream.

Because of this, the author proposes a Dual-Regime Principle:

Continuous Telemetry: Keep the black box running 24/7 to catch big changes and keep the chef honest most of the time.
Periodic Invasive Audits: Since the black box has a blind spot, you must physically open the robot's brain and inspect it regularly. The frequency of these inspections depends on how big the "blind spot" is. If the blind spot is small, you can wait longer between checks. If it's big, you need to check more often.

In short, the paper says we can't build a perfect, invisible fence around self-adapting AI. But we can build a very good fence with a few small gaps, and we can schedule regular patrols to walk through those gaps and make sure nothing bad is hiding there. It's a realistic, mathematically sound way to manage risk in a world where the tools are constantly changing themselves.

Technical Summary: Telemetry and Concealment in Self-Adapting Generative AI

1. Problem Statement

Current Model Risk Management (MRM) guidance assumes a static model lifecycle: models are developed, independently validated at a specific point in time, and deployed without further autonomous modification. Self-adapting generative AI systems (e.g., Large Language Models updated via Reinforcement Learning from Human Feedback or continuous Stochastic Gradient Descent) violate this assumption by design. These models update their own weights during production, rendering point-in-time validation inadequate and creating a "Continuous Validation Gap."

This paper addresses two interconnected governance challenges:

Observability: How to construct a rigorous, computationally tractable telemetry architecture that records the trajectory of a self-adapting model in both discrete and continuous time.
Adversarial Concealment: What happens when the model owner has an economic incentive to conceal learning updates to avoid triggering mandatory validation reviews? The paper formalizes the "Model Hiding Problem" and investigates the fundamental limits of detection.

2. Methodology

The paper is divided into two parts. Part I develops the telemetry architecture under the assumption of faithful reporting. Part II relaxes this assumption, modeling the interaction as an adversarial game between a model owner and a validator.

Part I: Telemetry Architecture

Discrete-Time Logging: The author proposes that weight-space distance (e.g., Frobenius norm) is an inadequate metric for audit. Instead, they define the Behavioral Divergence ( $\delta_t$ ) as the Kullback-Leibler (KL) divergence between the output distributions of consecutive model states on a canonical reference set $R$ . They construct a Merkle Weight Chain to cryptographically bind each model state to its predecessor and training data, ensuring tamper-evidence.
Event-Driven Logging: Rather than logging every step, the system logs only at Stopping Times defined by the crossing of a behavioral divergence threshold. This ensures log density is proportional to behavioral movement, not wall-clock time.
Continuous-Time Limit: The discrete SGD recursion is approximated by an Itô Stochastic Differential Equation (SDE). The author identifies Quadratic Variation $[W]_t$ as the canonical telemetry scalar for continuous time, capturing the stochastic intensity of learning.
Stability Framework: The architecture is embedded within a Lyapunov stability framework. The author derives conditions for both mean stability and almost-sure path-wise stability, utilizing the Law of the Iterated Logarithm (LIL) to bound martingale fluctuations.
Governance Trigger: A CUSUM (Cumulative Sum) test is proposed to operationalize tier-based governance escalation, providing a statistically grounded mechanism for detecting shifts in behavioral divergence with controlled false-alarm rates.

Part II: The Model Hiding Problem

The paper formalizes a game where the Model Owner ( $F$ ) attempts to maximize undetected drift while the Validator ( $V$ ) attempts to detect it. The author classifies six distinct attack strategies and analyzes countermeasures:

Discrete-Time Attacks:
- Attack S (Suppression): The firm stops transmitting telemetry. Countermeasure: A Behavioral Oracle using a secret canary set to detect output divergence.
- Attack F (Forgery): The firm reports a compliant "decoy" hash while serving a drifted model. Countermeasure: Inference Signing (binding outputs to model hashes) and Zero-Knowledge Proofs (zk-SNARKs) for Lyapunov compliance.
- Attack C (Shadow Model): The firm blends outputs from a reported model and an unreported side-channel model. Countermeasure: Trusted Execution Environment (TEE) attestation to bind the entire pipeline to a hardware-verified state.
Continuous-Time Attacks:
- Attack N (Diffusion-Amplified Drift Masking): The firm amplifies noise to hide drift. Countermeasure: The paper proves that Power Variation ratios fail to detect drift because the first variation of a diffusion is infinite; the ratio measures sampling frequency, not drift.
- Attack G (Girsanov Measure Manipulation): The firm manipulates training data to change the probability measure of the drift. Countermeasure: The proposed Gradient Probe is shown to be circular, as it only re-detects weight drift already visible to the Behavioral Oracle.
- Attack H (Hessian Exploitation): The firm engineers local curvature to cancel drift signals in mean-stability checks. Countermeasure: A Path-wise Stability Check using the LIL envelope to verify that realized paths stay within declared quadratic variation bounds.

Fundamental Limits

The paper establishes an Information-Theoretic Residual. Even with all countermeasures, there exists a strictly positive radius $\epsilon^*(\delta)$ of undetectable weight perturbations for any finite behavioral resolution $\delta$ . This implies that no telemetry-only regime can detect arbitrarily small adversarial drift.

3. Key Contributions

Minimal Sufficient Statistic: Formalization of the State Digest (behavioral divergence on a reference set) as the minimal sufficient statistic for discrete-time auditing, replacing weight-space distance.
Merkle Weight Chain: Construction of a tamper-evident chain binding model states to training data, reducing trust in history to the verification of a single chain tip.
Continuous-Time Generalization: Derivation of the continuous-time limit via Itô calculus, establishing Quadratic Variation as the canonical scalar for learning intensity and KL Stopping Times for event-driven logging.
Stability Certificates: Embedding the framework in Lyapunov theory to derive mean and almost-sure stability conditions, including the use of the Law of the Iterated Logarithm for path-wise bounds.
Model Hiding Taxonomy: A systematic classification of six concealment strategies (three discrete, three continuous) with formal countermeasures for each.
Negative Results: Rigorous proofs that natural continuous-time defenses (Power Variation and Gradient Probes) fail to detect specific adversarial drifts, highlighting the intrinsic limitations of telemetry.
Detection Floor Theorem: Proof that for any finite resolution $\delta > 0$ , the set of undetectable perturbations contains a ball of strictly positive radius $\epsilon^*(\delta)$ .
Dual-Regime Principle: A design recommendation that continuous telemetry is necessary but not sufficient, requiring pairing with periodic invasive audits to bound the undetectable leakage.

4. Results

The paper supports its theoretical claims with numerical studies (using a lightweight softmax classifier) and formal proofs:

Metric Efficacy: Study A1 demonstrates that equal-magnitude weight perturbations in the gradient direction produce significantly higher KL divergence (approx. 2.33 $\times$ ) than those in the tangent (null-space) direction, validating behavioral divergence over Frobenius norm.
Tamper Evidence: Study A2 confirms that a tiny modification ( $10^{-2}$ ) in weights invalidates the entire subsequent Merkle chain with overwhelming probability.
Event-Driven Logging: Study A3 shows that static models generate zero packets, while packet density scales with adaptation intensity, validating the efficiency of the stopping-time approach.
Quadratic Variation: Study A4 verifies that empirical quadratic variation matches the theoretical prediction $\eta \sigma^2 d_{tot} T$ within 0.56% error.
Stability and CUSUM: Study A5 confirms Lyapunov decay to a noise floor and demonstrates that the CUSUM trigger detects adversarial drift shortly after it begins while maintaining a low false-alarm rate.
Attack N Failure: Study B2 confirms that the Power Variation ratio is blind to drift, scaling only with the square root of the number of samples ( $\sqrt{n}$ ), regardless of drift magnitude.
Attack H Detection: Study B4 shows that while mean stability checks pass for an attacker under-declaring variance, the path-wise LIL envelope check detects the violation with a 99.7% rate.
Resolution Floor: Study B5 empirically demonstrates the existence of a finite-resolution gap where small perturbations remain undetected by the canary set, confirming Theorem 13.3.

5. Significance and Claims

The paper claims to provide the first formal treatment of governance evasion in the context of continually learning AI models. Its significance lies in:

Redefining Audit Metrics: Shifting the focus from weight-space distance to behavioral divergence, acknowledging that large weight moves can be behaviorally inert while small moves can be critical.
Bridging MRM and Continuous Learning: Adapting static MRM frameworks to dynamic systems by introducing continuous telemetry and stability certificates.
Honesty about Limits: Rather than claiming a perfect detection system, the paper rigorously defines the detection floor. It argues that because telemetry cannot detect arbitrarily small drifts, a Dual-Regime Principle is required: continuous telemetry narrows the scope of audits, but periodic invasive audits (under TEE) are strictly necessary to bound the residual risk.
Defense-in-Depth: The proposed architecture is not a single tool but a layered stack (Hardware, Cryptographic, Statistical, Stochastic, Governance) where each layer addresses specific attack surfaces.

The paper concludes that while continuous telemetry transforms governance from undirected snapshots to targeted investigations, it cannot replace invasive audits. The "dual-regime" approach is presented as the only structurally sound way to manage the risks of self-adapting AI within existing MRM guidance.