Building a house begins with a promise: a set of drawings that tells a builder exactly what to construct and how much material is needed. These blueprints are not mere sketches; they are complex instructions where a single line represents a wall, a note implies a specific type of wood, and a dimension string dictates the length of a beam. Turning these pages into a list of materials—counting every stud, every sheet of drywall, and every square foot of roofing—is a task known as a quantity takeoff. For decades, this has been the domain of human experts who must read, measure, and reconcile information scattered across dozens of pages, often filling in gaps with knowledge that never appears on the paper itself. A mistake here is not just a math error; it can mean thousands of dollars in wasted materials or a project that stalls because the wrong amount of concrete was ordered.

For years, computer scientists have tried to teach machines to do this work. Large language models, the powerful artificial intelligence systems that can write essays and answer questions, have been tested on these drawings. They can read the text and understand the words, but they struggle with the visual reality of the blueprint. They might see a wall but fail to measure it correctly, or they might count a door that appears on two different pages as two separate doors. They lack the specific, grounded understanding of construction conventions—like knowing that walls are built with studs spaced at a precise interval—that experienced builders carry in their minds. The result has been a gap between what machines can say and what they can actually calculate.

A team of researchers at Handoff AI Research has now introduced a system called Handoff-H1, designed to bridge this gap not by making a single, smarter machine, but by building a team of specialized tools that work together. Their work, evaluated against a new benchmark of real residential blueprints, shows that a system combining computer vision, specialized software tools, and a structured knowledge base can produce material estimates that are not only more thorough than those from current top-tier artificial intelligence models but also rival the accuracy and completeness of human experts.

The core of the problem lies in how construction drawings communicate. A blueprint does not list every single nail or stud; it shows the structure and relies on the reader to apply standard rules. If a drawing shows a wall, a human estimator knows to calculate the number of wooden studs based on a standard spacing, even if the drawing does not explicitly state the spacing. It requires looking at a floor plan, measuring a dimension, checking a note on a different page, and then applying a rule of thumb to arrive at a final number. This process involves three distinct challenges: seeing the visual elements clearly, reasoning across multiple pages to connect related information, and grounding the results in industry conventions that are never written down.

Handoff-H1 addresses these challenges by organizing the work into three layers. The first layer is a set of purpose-built computer vision models. Unlike general image readers, these models are trained specifically on construction drawings to identify and label the typed elements: rooms, walls, doors, and windows. They act as the eyes of the system, recovering the basic structure from the raw PDF files. The second layer is a persistent project foundation, a structured database that organizes what the eyes have seen. This foundation holds the information in a hierarchy that mirrors how construction projects are actually built, separating the work into different trades like framing, plumbing, and electrical. Crucially, this foundation is grounded in a curated knowledge base of construction rules and conventions, ensuring that the system knows, for instance, that a certain type of roof requires a specific calculation for slope that is not visible in the flat drawing.

The third layer is the orchestration of vision agents. These are software workers that use the information from the foundation and the tools provided by the vision models to perform the actual counting and measuring. They do not try to do everything at once. Instead, they focus on one trade at a time, using specialized tools to decompose complex plans into individual members, such as breaking a framing plan down into every single stud. If an agent is unsure, it can call upon a specialized tool to count or measure a specific region, rather than guessing. Finally, an independent verification pass audits the work, checking for missed items or impossible quantities before the final list is produced.

To test this system, the researchers created a benchmark called TAKEOFFBENCH-V1. They gathered ten sets of real, permissioned residential blueprints and paired them with a "gold standard" list of materials. This gold standard was not created by a single person but was the result of a consensus process where multiple independent professional estimators worked on the same drawings and their results were reconciled to create a single, verified truth. This approach acknowledges that even human experts can disagree on certain measurements, so the benchmark reflects a realistic, high-quality standard rather than an impossible ideal. The benchmark includes over two thousand verified line items, covering nine different construction trades, from concrete and framing to drywall and roofing.

When the researchers ran the test, the results were striking. They compared Handoff-H1 against seven of the most advanced artificial intelligence models available, including both closed and open-weight systems. These models were given the same raw PDF files and asked to produce a list of materials. The best of these frontier models achieved a composite score of roughly 61 percent. In contrast, Handoff-H1 reached a score of 81.6 percent. This gap of about twenty points is significant, suggesting that the specialized architecture of Handoff-H1—its combination of vision, knowledge, and orchestration—solves problems that general-purpose models cannot.

Perhaps more importantly, the system was compared directly to independent professional estimators, the human experts who created the ground truth. When these humans were scored against the same gold standard, they achieved a composite score of 77.6 percent. Handoff-H1 outperformed the human average, reaching a higher score than the experts in six of the ten drawing sets. The system did this by being more thorough; it found and counted more items than the humans did, particularly in complex areas like framing and roofing where human estimators often miss details or skip scopes. While the human experts were slightly more precise on the items they did find, the system's ability to cover more ground resulted in a better overall estimate.

The analysis of the results revealed where the system excels and where it still faces challenges. The system performed exceptionally well on trades that require deriving quantities from areas and slopes, such as roofing and drywall, where human estimators often struggle with consistency. It also performed very well on counting tasks like plumbing fixtures and windows. However, the system, like the humans, found continuous measurements—such as the exact length of a wall or the area of a roof—to be the most difficult. These tasks require resolving ambiguities in the drawings, such as whether a dimension line includes the thickness of a wall, a debate that even human experts can have. The system's precision on these measurements was slightly lower than that of the human experts, but its coverage was far superior, meaning it missed fewer items overall.

The researchers emphasize that this success is not due to a single breakthrough in artificial intelligence, but rather the careful combination of different capabilities. General-purpose models, even the most advanced ones, failed to reach the level of the human experts or the Handoff-H1 system when given the same raw input. They tended to produce estimates that looked fluent but lacked dimensional grounding, often hallucinating quantities or missing the trade-specific logic required to turn a drawing into a material list. The Handoff-H1 system succeeded because it did not rely on a single model to do everything; instead, it used a structured approach that separated seeing, knowing, and reasoning.

This work suggests a path forward for artificial intelligence in specialized fields. It demonstrates that for complex, real-world tasks, the most effective systems may not be the ones with the largest language models, but those that are built with a deep understanding of the domain's structure and rules. By combining computer vision with a curated knowledge base and a structured workflow, the system was able to achieve a level of performance that rivals human experts. The researchers have made their evaluation tools public, allowing others to test their own systems against the same benchmark, while keeping the actual blueprint data and ground truth restricted to prevent the system from simply memorizing the answers. This approach ensures that future progress will be measured by genuine capability rather than by the ability to recall training data.

The study concludes that while the system is not perfect, it represents a significant step toward automating a task that has long been the exclusive domain of human experts. It does not replace the need for human oversight, but it offers a tool that can be more thorough and consistent than a human working alone. In a field where a single error can cost thousands of dollars, the ability to generate a more complete and accurate list of materials from a set of drawings is a tangible improvement. The system's performance, surpassing both the best general artificial intelligence models and the average human estimator, indicates that the combination of specialized vision, structured knowledge, and orchestrated reasoning is a powerful approach to solving complex, real-world problems.