Seven simple steps for log analysis in AI systems

This paper proposes a standardized, seven-step pipeline for analyzing AI system logs to ensure rigorous and reproducible evaluation, complete with implementation guidance in the Inspect Scout library and examples of common pitfalls.

The Gist

Imagine you've built a super-smart robot assistant. You've given it a complex job, like solving a cybersecurity puzzle or writing a novel. You press "Go," and the robot gets to work. But when it's done, you don't just get a final grade; you get a gigantic, messy notebook filled with every thought the robot had, every tool it tried to use, every mistake it made, and every time it got confused.

This notebook is called a Log.

The paper you're asking about is essentially a guidebook on how to read that messy notebook without going crazy. It's titled "Seven Simple Steps for Log Analysis in AI Systems," and it's written by a team of experts who want to help researchers understand what their AI is actually doing, rather than just guessing.

Here is the guide, translated into everyday language with some fun analogies.

---

The Big Picture: Why Do We Need This?

Think of an AI system like a black box. You put a question in, and an answer comes out. But inside that box, the AI is doing thousands of tiny things: thinking, checking its work, calling a calculator, maybe even getting stuck in a loop.

If you just look at the final answer, you might miss the fact that the AI was actually hallucinating (making things up) or refusing to do the job because it was scared. Log analysis is like taking the black box apart and looking at the gears turning. It turns a chaotic pile of data into a clear story.

The paper says, "We don't have a standard way to do this yet, so here is a recipe (a pipeline) to follow."

---

The 7 Steps (The Recipe)

Step 1: Define Your Mission (The "Why")

Before you open the notebook, you need to know what you are looking for.

• Analogy: Imagine you are a detective arriving at a crime scene. Do you want to know who did it? How they did it? Or why they did it?
• The Paper Says: Don't just look at the logs randomly. Ask a specific question first. "Did the robot refuse to do a dangerous task?" or "Did the robot get stuck in a loop?" This helps you focus your search.

Step 2: Organize the Evidence (The "Database")

Logs are messy. They come from different robots, different times, and different tasks. You need to sort them.

• Analogy: Imagine you have a mountain of loose puzzle pieces from 50 different puzzles. Before you can solve anything, you need to sort them into boxes by color and shape.
• The Paper Says: Clean up your data. Throw away incomplete runs (like a puzzle with missing pieces). Make sure the labels are consistent so you can search for them later.

Step 3: Take a Peek (The "Exploration")

Now, look at a few pages of the notebook manually. Don't try to read everything yet; just get a feel for the vibe.

• Analogy: You're a librarian walking through the stacks. You pull out a few random books to see if the pages are torn, if the ink is smudged, or if the stories make sense. You're looking for "red flags."
• The Paper Says: Read a few examples. Look for weird patterns. Did the robot say "I can't do that" a lot? Did it get stuck on the same error? This helps you guess what to look for in the big data.

Step 4: Sharpen Your Question (The "Refinement")

Based on what you saw in Step 3, make your question more specific.

• Analogy: You started by asking, "Is the robot acting weird?" After looking at a few pages, you realize, "Oh, it's not acting weird; it's specifically refusing to use the internet." Now your question is sharper.
• The Paper Says: Turn your vague idea into a concrete signal. Instead of "Is it bad?", ask "Does it use the phrase 'I cannot' when asked to hack a website?"

Step 5: Build a Robot Detective (The "Scanner")

You can't read millions of pages by hand. You need to build a tool (a "scanner") to do it for you. This tool can be a simple search function or a smart AI that reads the logs.

• Analogy: You hire a metal detector to walk through the beach for you. You tell it, "Only beep if you find gold." You have to teach the detector exactly what gold looks like so it doesn't beep for every piece of trash.
• The Paper Says: Write a program (or a prompt for another AI) that scans the logs for your specific signal. Give it clear rules: "If the robot says 'sorry' and then stops, mark it as a 'Refusal'."

Step 6: Test Your Detective (The "Validation")

Your new robot detective might be too sensitive (barking at everything) or too lazy (missing the gold). You need to test it.

• Analogy: You take your metal detector out for a test run. You bury some gold and some trash in the sand. Does the detector find the gold? Does it ignore the trash? If it beeps at a soda can, you need to adjust the settings.
• The Paper Says: Compare your scanner's results against human experts. If the scanner says "Refusal" but a human says "No, that was just a pause," you need to fix your scanner's rules.

Step 7: Use the Findings (The "Action")

Now that you have clean, reliable data, what do you do with it?

• Analogy: You found out that your robot refuses to work on Tuesdays. Now you can either fix the robot (change its programming) or warn the users (tell them not to use it on Tuesdays).
• The Paper Says: Use the data to improve the AI, fix the evaluation tests, or publish your findings so others learn from them. Don't just guess; use the numbers.

---

The "Secret Sauce" (Key Takeaways)

• Don't Trust, Verify: Just because the AI gave an answer doesn't mean it did the work correctly. The logs tell the real story.
• Context is King: A robot saying "I can't do that" might be a safety feature (good!) or a bug (bad!). You need to know the context to tell the difference.
• Iterate, Iterate, Iterate: You won't get the perfect scanner on the first try. You'll build it, test it, break it, fix it, and test it again. That's normal!
• Tools Matter: The authors mention a tool called Inspect Scout. Think of this as a "Swiss Army Knife" for log analysis that helps you organize, search, and scan your data easily.

The Bottom Line

This paper is a manual for AI detectives. It teaches us how to stop guessing what our AI models are thinking and start knowing by systematically reading their "diaries" (logs). By following these seven steps, we can build safer, smarter, and more reliable AI systems.

Technical Summary

1. Problem Statement

As AI systems (specifically agents and chatbots) become more capable and operate in complex, multi-turn environments, they generate massive volumes of unstructured logs. These logs contain critical data regarding model capabilities, behaviors, reasoning traces, tool usage, and safety failures. However, the field lacks a standardized, reproducible methodology for analyzing these logs.

• Current Challenges: Researchers often rely on ad-hoc methods, manual inspection, or informal blog posts rather than systematic pipelines.
• The Gap: While automated scanners (LLM-as-a-Judge) exist, there is no unified framework to guide researchers from raw log collection to rigorous statistical analysis. This hinders the ability to detect subtle patterns (e.g., refusal behaviors, evaluation awareness, reward hacking) at scale and compromises the validity of AI evaluations.

2. Methodology: The 7-Step Pipeline

The authors propose a structured, seven-step pipeline for log analysis, illustrated with concrete code examples using the open-source Inspect Scout library. The pipeline is designed to be applicable to both exploratory analysis (existing logs) and hypothesis-driven analysis.

Step 1: Define the Purpose

• Goal: Establish the research question (primary or secondary).
• Action: Distinguish between questions about the AI system (e.g., "Does it refuse harmful requests?") and the evaluation environment (e.g., "Are tools unavailable?").
• Context: Requires understanding the task setup, model configuration (temperature, token limits), agent scaffolding, and environment constraints.

Step 2: Prepare Database of Logs

• Goal: Organize raw logs into a structured, queryable database.
• Action: Filter incomplete runs, remove PII, standardize metadata (timestamps, scores), and enrich logs with additional context (e.g., solution write-ups).
• Tool: Uses frameworks like Inspect AI to structure logs immediately upon collection.

Step 3: Explore Logs

• Goal: Gain intuition about data structure and content before automated analysis.
• Action:
  • Metadata Exploration: Analyze token counts, model names, and success scores.
  • Manual Sampling: Read transcripts strategically (e.g., sampling by score, length, or error types) to identify failure modes.
  • Automated Exploration: Use string matching for obvious errors or LLMs (via RAG or direct querying) to detect complex patterns and generate hypotheses.

Step 4: Refine the Research Question

• Goal: Transform abstract questions into concrete, measurable signals.
• Action: Define specific "signals" (e.g., specific keywords, tool errors, reasoning patterns) that can be operationalized into scanners.
• Differentiation: Distinguish between environment-focused signals (e.g., "command not found") and agent-focused signals (e.g., "sycophancy," "hallucination").

Step 5: Develop Scanner

• Goal: Build an automated detector (scanner) for the defined signals.
• Design Decisions:
  • Chunking: Decide whether to analyze the whole transcript or specific parts (e.g., tool calls only).
  • Scoring: Choose output types (Binary, Multi-class, Counts, Ordinal, Pairwise).
  • Best Practices: Use detailed rubrics, provide examples (anchors), include "other" categories, and request confidence scores/explanations from the LLM grader.
• Implementation: The paper emphasizes using LLMs as judges but warns against them performing the task rather than grading it.

Step 6: Validate Scanner

• Goal: Ensure the scanner accurately detects the intended patterns.
• Action:
  • Stratified Sampling: Create a validation set covering different outcomes, uncertainty levels, and classes.
  • Ground Truth: Obtain labels via human annotators (blind to scanner outputs) or objective measures.
  • Metrics: Calculate Precision, Recall, F1, ROC-AUC, or calibration scores depending on the task.
  • Iterative Refinement: Use validation results to refine the prompt and rubric, addressing edge cases and biases (e.g., length bias, self-bias).

Step 7: Use Results

• Goal: Interpret findings and apply them.
• Application:
  • Flagging: Deploy scanners in production to trigger immediate actions (e.g., blocking harmful outputs).
  • Research: Convert unstructured logs into structured datasets for statistical analysis (e.g., Hierarchical Bayesian GLMs) to draw generalizable conclusions about model behavior.
  • Rigor: Avoid cherry-picking; use systematic analysis and triangulation of signals.

3. Key Contributions

1. Standardized Framework: The paper provides the first comprehensive, step-by-step pipeline for log analysis in AI, moving the field from ad-hoc practices to a rigorous scientific standard.
2. Open-Source Tooling (Inspect Scout): The authors demonstrate the pipeline using Inspect Scout, an open-source library that supports log exploration, scanner building, and validation, making the methodology immediately actionable for researchers.
3. Detailed Best Practices: It offers specific, actionable guidance on:
   • Designing LLM-based scanners (rubrics, chunking, scoring types).
   • Validating scanners against human ground truth.
   • Avoiding common pitfalls (e.g., scanner bias, overfitting to specific cases).
4. Signal Taxonomy: The paper categorizes specific signals relevant to agents (e.g., tool errors, evaluation awareness) and chatbots (e.g., sycophancy, crisis response), aiding in the identification of what to look for.
5. Identification of Open Questions: It highlights critical gaps in current knowledge, such as optimal sample sizes for validation, the impact of transcript length on scanner reliability ("lost in the middle" effect), and the best methods for detecting abstract behaviors.

4. Results (Illustrative Case Study)

The paper does not present a single new scientific discovery but rather validates its framework through a case study on Cybench (a cybersecurity benchmark).

• Scenario: The authors analyzed logs from agents attempting to solve Capture The Flag (CTF) tasks.
• Discovery: Initial manual exploration revealed that models were frequently refusing tasks or abandoning them due to perceived danger, which skewed the evaluation scores.
• Application:
  • They refined their research question to focus on "refusal behaviors."
  • They built a multi-class scanner (No Refusal, Partial, Indirect, Critical) using Inspect Scout.
  • They validated the scanner against a human-labeled set, achieving a high F1 score (0.998 for binary classification).
  • Outcome: The refined scanner successfully identified subtle refusal patterns (e.g., agents claiming tasks were unsolvable) that simple keyword matching missed, allowing for a more accurate assessment of model capabilities.

5. Significance

• Reproducibility: By standardizing the log analysis process, the paper enables researchers to reproduce studies and compare results across different models and evaluations.
• Safety and Alignment: Robust log analysis is critical for detecting safety failures (e.g., reward hacking, deception, refusal to follow safety guidelines) that might be missed in aggregate metrics.
• Scalability: The framework enables the transition from manual, small-scale log review to automated, large-scale analysis, which is essential as AI systems become more autonomous and complex.
• Community Building: The paper serves as a foundational guide for the AI safety and evaluation community, encouraging the sharing of methodologies and the systematic investigation of open questions.

In summary, this paper acts as a "how-to" manual for the AI research community, transforming log analysis from a chaotic, manual task into a rigorous, automated, and scientifically valid discipline.