🤖 AI

EchoCoT: Extracting Hidden Chain-of-Thought from Large Reasoning Models

This paper introduces EchoCoT, a multi-step attack framework that successfully extracts hidden chain-of-thought traces near-verbatim from both open-source and frontier proprietary large reasoning models via API interactions, demonstrating a significant security risk to these valuable model assets.

Original authors: Yiting Qu, Ziqing Yang, Chi Cui, Ye Leng, Junjie Chu, Yang Zhang

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiting Qu, Ziqing Yang, Chi Cui, Ye Leng, Junjie Chu, Yang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems have become remarkably skilled at solving complex problems, from writing computer code to navigating scientific puzzles. When these advanced models tackle a difficult question, they do not simply spit out a final answer. Instead, they first generate a long, internal stream of thought, a step-by-step reasoning process where they explore ideas, check their work, correct mistakes, and consider different paths before settling on a conclusion. This internal monologue, often called a chain of thought, is usually hidden from the user. The model keeps this reasoning private, showing only the final result to protect its intellectual assets and prevent the exposure of sensitive data or proprietary logic. For developers, this hidden reasoning is a valuable resource for improving future models, while for users, it is a black box that promises a correct answer without revealing the messy work behind it.

A team of security researchers at CISPA Helmholtz Center for Information Security has discovered that this hidden reasoning is not as secure as once believed. They found a way to trick these advanced models into revealing their entire internal thought process, almost word for word, even when the models are designed to keep it secret. The researchers developed a method called EchoCoT, which exploits a specific quirk in how these models interact with external tools. In many systems, when a model needs to use a tool—like a calculator or a database—it pauses its reasoning to make the call, but it keeps its previous thoughts active in its memory to ensure continuity. The researchers realized that an attacker could use this continuity to their advantage. By repeatedly asking the model to save its thoughts to a tool and then rejecting those saved thoughts as incomplete, they could force the model to replay its internal reasoning, each time revealing more detail until the full, hidden chain of thought was exposed.

The researchers tested this technique on a variety of models, including both open-source systems and the most advanced, proprietary models from major technology companies. On the open-source models, where the researchers could compare the extracted thoughts against the original hidden ones, they found that EchoCoT could recover nearly the entire reasoning process. In many cases, the extracted text matched the original hidden thoughts with extreme precision, capturing over 90 percent of the exact words and phrases. The method worked even for very long reasoning chains, successfully extracting thousands of tokens of thought. When the researchers applied the same technique to the proprietary models from companies like Google and Anthropic, they could not see the original hidden thoughts to compare them directly. However, the extracted thoughts were incredibly long, often matching the length of the reasoning the companies reported the models had used. Furthermore, the content of these extracted thoughts aligned closely with the brief summaries the companies provided, suggesting the extraction was accurate. In one instance involving a Google model, the researchers recovered a reasoning chain of over 33,000 tokens, which was nearly identical in length to the model's reported internal process.

Beyond just recovering the text, the extracted thoughts revealed rich details about how these models think. The researchers observed the models simulating Google searches, switching between different languages like Chinese and English while recalling information, and engaging in extensive self-correction. These behaviors, such as a model saying "Wait, is this true?" or "Let me try a different approach," are part of the raw internal process that is typically stripped away before the final answer is shown to a user. The study also showed that the attack could be automated. The researchers built a system that could learn the best way to ask for the thoughts, refining its questions based on feedback signals like the number of words the model used. This automated approach made the attack even more effective, allowing it to generalize across different types of questions and datasets without needing to be manually reprogrammed for each new problem.

The implications of this discovery are significant for the security of artificial intelligence. The researchers found that simply hiding the reasoning from the user is not enough to protect it. As long as the model retains its internal state to perform tasks that require tool use, that state can be replayed and extracted. The study tested several potential defenses, such as hiding the count of reasoning words or adding strict instructions to the model's system prompt to prevent leakage. While some of these measures reduced the success of the attack, none completely stopped it. The most effective defense tested was to remove the model's access to its own previous reasoning after a tool call, but this would break the model's ability to perform complex, multi-step tasks. The researchers responsibly reported their findings to the major technology companies involved, highlighting a critical vulnerability in the current design of these powerful reasoning systems. Their work suggests that protecting the intellectual property and safety of these models will require more fundamental changes to how they are built and how they interact with the outside world, rather than just relying on the assumption that hidden thoughts remain hidden.

=== SUMMARY ===

Technical Summary: EchoCoT – Extracting Hidden Chain-of-Thought from Large Reasoning Models

1. Problem Statement

Large Reasoning Models (LRMs) such as GPT-5, DeepSeek-R1, and Gemini Thinking generate internal "hidden" chains-of-thought (CoT) before producing a final answer. These traces contain rich information, including step-by-step calculations, failed attempts, and self-corrections, making them valuable assets for model training, distillation, and safety monitoring. Consequently, frontier providers strictly protect these traces from user exposure.

While prior research has focused on inferring sensitive user data from hidden CoTs, the question of whether the hidden CoTs themselves can be extracted verbatim from black-box LRMs via API interactions remains largely unexplored. Existing attempts, such as few-shot prompting (REP) or replaying encrypted reasoning blocks (Stolen Thoughts), either fail to achieve near-verbatim recovery or rely on architecture-specific vulnerabilities (e.g., reusable encrypted blocks) that are not universally applicable. The core challenge is extracting the hidden reasoning $c^*$ from a black-box model $M(x, t) \to (c^*, y^*)$ without access to internal states, parameters, or the ground-truth trace.

2. Methodology: EchoCoT

The authors propose EchoCoT, a multi-step attack that exploits a previously overlooked "reasoning replay surface" created by tool-call interactions.

2.1 The Replay Surface

In standard multi-turn conversations, providers typically discard hidden reasoning tokens after each turn to save context. However, to maintain reasoning continuity during tool calls (function calls), many providers preserve the hidden CoT within a single user-assistant turn across multiple tool invocations. EchoCoT leverages this by:

Injecting a Scratchpad Tool: The attacker defines a tool that allows the model to "archive" its reasoning.
Iterative Extraction: The attacker initiates a tool call, prompting the model to archive its reasoning. The initial archive is often highly compressed. The attacker then injects a malicious instruction (via the tool response) rejecting the partial archive and demanding a more complete reproduction of the internal thinking block.
State Preservation: Because the interaction remains within a single turn, the model retains the original hidden CoT in its context, allowing it to "replay" and expand the trace in subsequent tool arguments.

2.2 Online Fidelity Evaluation

Since the attacker cannot observe the ground-truth CoT, EchoCoT uses proxy signals provided by the API to evaluate extraction quality:

Reasoning Token Count: The API reports the number of tokens used for reasoning ( $|c^*|$ ). The attacker compares this with the token count of the extracted candidate ( $|\hat{c}|$ ) to compute Length Error.
CoT Summary: Some APIs provide a compressed summary of the reasoning. The attacker computes Summary Token Recall to measure how well the extracted candidate covers the semantic content of the summary.

2.3 Automated Injection Optimization

To avoid manual trial-and-error, the authors develop an LLM-based optimization framework that searches for a universal injection trajectory ( $P = p_1, \dots, p_K$ ) across diverse questions. The framework operates in three stages:

INJECT: The optimizer generates injections step-by-step for a batch of questions, perturbing dimensions like authority framing, rejection wording, and completeness requirements based on previous fidelity scores.
REFLECT: After processing a batch, the optimizer analyzes aggregate metrics (e.g., which steps improved fidelity, which caused refusals) to diagnose the current trajectory.
DISTILL: Lessons learned are distilled into an "experience file" (working memory) to guide future batches, preventing redundant exploration.

The optimization uses two objectives:

Length-Guided Optimization (LGO): Maximizes length fidelity (minimizing Length Error).
Length and Text-Guided Optimization (LTGO): Jointly optimizes for length and textual fidelity (using Summary Token Recall) when summaries are available.

3. Key Contributions

Identification of a Replay Surface: The paper identifies that tool-call interactions preserve hidden reasoning states across multiple calls within a single turn, creating a vulnerability for CoT extraction.
EchoCoT Attack: A multi-step attack framework that iteratively induces LRMs to reveal hidden CoTs near-verbatim by exploiting this replay surface and using API signals for feedback.
Automated Optimization Framework: An LLM-driven system that automatically discovers effective, universal injection trajectories for specific target models without requiring ground-truth CoTs during the search process.
Extensive Evaluation: A systematic assessment of CoT extraction risks across open-source and frontier proprietary LRMs, demonstrating the feasibility of near-verbatim extraction.

4. Experimental Results

4.1 Open-Source LRMs

Evaluated on DeepSeek-V4-Flash, Qwen3.5-Plus, and GLM-5.2 using the OpenThoughts dataset:

Near-Verbatim Recovery: EchoCoT-LTGO achieved an Attack Success Rate (ASR@90) of up to 66.4% on DeepSeek-V4-Flash (requiring Length Error $\le$ 0.1 and Token-EM $\ge$ 0.9).
Fidelity: Extracted traces matched the target length within 10% and achieved high Token-EM scores (e.g., 0.999 in a 21k-token example).
Generalization: Trajectories optimized on OpenThoughts generalized to unseen datasets (MATH500, JEEBench, LiveCodeBench), achieving up to 80% ASR@90 on MATH500 for DeepSeek.
Length Robustness: The method successfully extracted traces exceeding 20,000 tokens.

4.2 Frontier Proprietary LRMs

Evaluated on Gemini-2.5, Gemini-3.1, Gemini-3.5, Sonnet-4.6, and Opus-4.6:

Long Trace Extraction: EchoCoT recovered traces closely matching provider-reported lengths. For example, on Gemini-2.5, it extracted 33,463 tokens from a target of 32,948 tokens.
Semantic Alignment: Extracted CoTs showed high semantic alignment with provided summaries (Entailment Scores up to 0.994) and significant token overlap.
Rich Internal Behaviors: Extracted traces revealed internal behaviors not seen in final outputs, such as simulated Google searches, language switching (e.g., English to Chinese), and self-correction.
System Prompt Leakage: In one instance targeting o4-mini, the attack exposed system-prompt content matching public reports, suggesting leakage may extend beyond CoTs.

4.3 Defense Evaluation

The authors evaluated several defenses:

Reasoning State Removal: Removing the hidden CoT after the first tool call completely eliminated the attack (0% ASR), but is impractical as it breaks task continuity.
System Prompts: Adding instructions to treat tool outputs as untrusted reduced ASR significantly (from 46% to 5% mean), but adaptive attackers could still achieve ~10%–29% success.
Length Signal Obfuscation: Hiding or perturbing the reasoning token count reduced success rates, but adaptive attackers using total token usage estimates could partially recover performance.
Conclusion: No single practical defense fully mitigates the risk; a multi-layered approach is necessary.

5. Significance and Claims

The paper argues that hidden CoT extraction is a practical security risk. The authors claim that:

Hiding CoTs is Insufficient: Simply not displaying CoTs to users does not prevent extraction if the model retains reasoning continuity during tool calls.
API Signals are Vulnerable: Seemingly benign signals like reasoning token counts and CoT summaries provide sufficient feedback for an attacker to iteratively refine and recover near-verbatim traces.
Scalability of Risk: The ability to automatically optimize injection trajectories makes the attack scalable across different models and datasets.

The work highlights the need for better protection of hidden CoT assets, suggesting that future defenses must address reasoning continuity, system-level instructions, and API signal exposure simultaneously. The authors responsibly disclosed these findings to major providers (Google, Anthropic, OpenAI) and emphasize that the study aims to assess exposure risks rather than to exploit proprietary models for practical gain.