Failure Taxonomy of an Agent-Run Research Lab: 105 Documented Process Failures, Who Caught Them, and an Open Dataset

In the modern landscape of artificial intelligence, a new kind of laboratory has emerged where the primary workers are not humans, but software agents. These are autonomous programs designed to perform complex tasks like writing code, analyzing data, or drafting reports, often working in fleets where one agent builds a project while another acts as a critic to find errors. For years, the focus of research in this field has been on how well these agents perform their final tasks: did the code run? Did the model answer the question correctly? However, a critical blind spot remained. Scientists knew very little about the internal machinery of the research process itself. They did not have a clear map of where these automated teams stumble, what kinds of mistakes they make most often, or, perhaps most importantly, who or what catches those mistakes before they cause a failure. Understanding the specific ways these digital workflows break down is essential for building systems that are not just powerful, but reliable and safe.

A recent technical report from a research group known as 321 AI Labs addresses this gap by turning its own internal operations into a case study. Instead of looking at the final output of a single project, the researchers spent two weeks in September 2026 meticulously documenting every single failure that occurred within their own agent-run research lab. They collected 105 distinct records of things that went wrong, ranging from a local software project to a job-application agent. Each record was treated as a piece of evidence, pinned to a specific log entry or file, creating a sealed, machine-readable dataset. The goal was not to hide these errors, but to categorize them with the same rigor a biologist might use to classify a new species of insect. By sorting these failures into specific groups based on where they happened in the workflow, why they happened, and who found them, the team created the first open dataset of its kind, offering a rare, unvarnished look at the operational reality of autonomous research.

The analysis of these 105 records revealed a clear pattern in how these automated teams operate. The failures were not scattered randomly; they clustered heavily in the phases where work is built and checked. The most common type of error, accounting for nearly half of all incidents, was a "verification gap." This is a specific kind of mistake where a safety check or a test existed on paper but was never actually run or connected to the system. It is the digital equivalent of having a fire extinguisher mounted on the wall that no one ever checks to see if it works. The second and third most common causes were environmental shifts, where the conditions the software ran in changed unexpectedly, and specification drift, where the project's goals slowly wandered away from the original plan. Interestingly, the data showed that the fleet's catching work was primarily performed by the agents themselves. In 69 percent of the cases (72 out of 105), the failure was caught by the agent that generated it (labeled "self_caught"). In the remaining 31 percent, a separate reviewing agent found the mistake. Crucially, the report found zero instances where a failure slipped through all layers of defense and escaped detection entirely during this period. Furthermore, zero instances required the "owner" layer to step in to catch a failure, indicating that the layers below the owner successfully handled the documented issues within this window.

The researchers did not simply count these errors; they subjected their own classification system to a strict audit to ensure they were all agreeing on what they were seeing. Two independent classifiers, working blindly without knowing each other's choices, labeled a sample of the records. Initially, they disagreed on roughly half of the decisions regarding where a failure occurred in the process. Through a series of revisions to their definitions and rules, they refined their understanding until their agreement became nearly perfect. This process of auditing their own labels is a significant part of the report's contribution, proving that the categories used to describe these failures are stable and reliable. The team also made a point to report what the data could not tell them. While they tracked the types and locations of errors, the vast majority of the records contained no information about the cost of fixing them. Because the source documents did not state a cost, the researchers refused to guess or estimate, leaving those figures blank. This adherence to "negative results"—reporting that something is unknown rather than inventing a number—was a deliberate choice to maintain the integrity of the dataset.

The value of this work lies in its transparency and its utility for future governance. The dataset is now available for other researchers to use, with a clear format that allows them to add their own failure records from different labs. This creates a growing census of process failures that can help the entire field understand where automated systems are fragile. The report suggests that the biggest opportunity for improvement is not in making agents smarter, but in fixing the "verification gaps" where checks exist but are not exercised. By knowing exactly where the failures concentrate, lab operators can direct their attention to the specific parts of the pipeline that need better monitoring. The study concludes that while these agent-run labs are capable of catching their own mistakes, the system is currently held together by a high volume of self-correction and review, and that the most common failure is not a lack of intelligence, but a failure to run the tests that were already designed. This shift from guessing about system reliability to measuring it with hard data marks a maturing step for the field of autonomous research.