🤖 AI

TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models

The paper introduces TreeAgent, a generalizable multi-agent framework that combines Vision-Language Models with compiled expert decision rules to automate and scale bias labeling in forestry remote sensing, significantly reducing expert annotation costs while maintaining interpretability and accuracy.

Original authors: Shiyi Chen, Nicholas Saban, Collin Hargreaves, Huiqi Wang

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiyi Chen, Nicholas Saban, Collin Hargreaves, Huiqi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to measure the height of millions of trees in a forest to calculate how much carbon they store. This is crucial for climate policy, but it's a messy job. You have three different ways to measure a tree:

The Field Crew: A human walks up with a tape measure (but they can't always see the very top of the tree if leaves are in the way).
The Airborne Laser: A plane shoots lasers at the trees (but if the laser pulses are too far apart, it might miss the tip).
The Computer Model: A software program guesses the height based on the laser data (but it can get confused by shadows or grid lines).

These three methods often disagree. Sometimes the human is wrong, sometimes the laser is wrong, and sometimes the computer is wrong. To fix this, forestry experts have to manually look at each tree and decide why the measurements disagree. They have a specific rulebook with seven different "bias" categories (e.g., "The human measured too low," or "The computer missed the top").

The Problem:
This manual checking is incredibly slow. It takes an expert about 5 minutes per tree. If you have a forest with millions of trees, this task is impossible to finish. Also, different experts might disagree with each other, making the "gold standard" data noisy.

The Solution: TreeAgent
The researchers built a new system called TreeAgent. Think of it not as a single "smart" AI trying to guess the answer, but as a team of specialized workers following a strict, pre-written instruction manual.

Here is how the team works, using a creative analogy:

1. The "Rulebook" (The Decision Tree)

Imagine a giant, complex flowchart (a decision tree) that an expert wrote down in plain English.

Example Rule: "If the difference between the laser height and the human height is less than 2%, mark it as 'No Difference.' Otherwise, check the ground."

Usually, if you want a computer to follow a rulebook, you have to hard-code it. If the expert changes their mind about the "2%" rule later, you have to hire a programmer to rewrite the code. That's slow and expensive.

TreeAgent's Innovation: They created a "Universal Translator" (called the Neural Rule Transpiler).

The expert just writes the rule in plain English.
The Translator instantly converts that sentence into a structured, executable flowchart.
The Magic: If the expert changes the rule later (e.g., "Change 2% to 3%"), you just update the sentence. The system automatically re-draws the flowchart. You don't need to touch the code. It's like changing the recipe in a cookbook without having to rebuild the kitchen.

2. The "Workers" (The Multi-Agent System)

Once the flowchart is built, the system runs through it like a detective solving a case.

The Calculator: For simple math (e.g., "Is the difference less than 2%?"), a fast, deterministic calculator does the math. No guessing, no errors.
The Visual Detective (VLM): Sometimes the numbers aren't enough. The flowchart might say, "Look at the picture: Is the tree's crown overlapping with its neighbor?"
- Here, the system calls in a Vision-Language Model (VLM). This is an AI that can "see" images and read text.
- To make sure the AI doesn't get confused or hallucinate, the system asks three different AI agents to look at the same picture and vote. If 2 out of 3 say "Yes, they overlap," the system accepts it.

3. The Results

The researchers tested this on a dataset of trees from different forests.

Speed: The old way took 5 minutes per tree. TreeAgent takes 0.04 minutes (about 2.4 seconds) per tree. That's roughly 125 times faster.
Accuracy:
- A standard computer learning model (trained on data tables) got about 36% of the classifications right.
- TreeAgent got 67.6% right.
- While it's not perfect yet, it is significantly better than the standard computer models and much faster than humans.

Why This Matters

The paper argues that for complex scientific jobs where experts have clear rules but need to look at pictures occasionally, this "Team + Rulebook" approach is the sweet spot.

Pure AI (trying to learn everything from scratch) is often a "black box"—you don't know why it made a decision, and it can be unreliable.
Pure Rules (hard-coded logic) are rigid and hard to update.
TreeAgent combines the best of both: it follows the expert's logic strictly (so you know why it made a decision) but uses AI only for the parts that require "eyes" (looking at the images).

The Catch:
The system is only as good as the rulebook. If the expert's rule is slightly off, the system follows it faithfully. Also, the "Visual Detective" (the AI looking at the pictures) still makes mistakes, especially with tricky forest images it hasn't seen before. But by using a voting system, they minimized these errors.

In short, TreeAgent is a fast, transparent, and updatable assistant that lets forestry experts scale their work from a few trees to millions, without losing the ability to explain why a tree was labeled a certain way.

Technical Summary: TreeAgent – A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry

1. Problem Statement

In forestry remote sensing, accurate tree height estimation is critical for terrestrial carbon accounting and climate policy. However, three primary data sources—field measurements, airborne LiDAR point clouds, and LiDAR-derived Canopy Height Models (CHMs)—systematically disagree, exhibiting distinct error modes. Correcting these biases requires domain experts to manually label individual trees based on complex diagnostic rules that distinguish seven bias types. This process is slow (3–5 minutes per tree), inconsistent across annotators, and does not scale to continental inventories. Furthermore, the "reference" data generated by these experts often contains noise due to inter-expert variability, creating a bottleneck for training supervised Machine Learning (ML) models.

The core challenge is to automate this labeling process while maintaining the interpretability and fidelity of expert-defined rules, without relying on end-to-end deep learning models that discard structural logic or single-prompt Large Language Models (LLMs) that lack reliability.

2. Methodology

The Decoupled Declarative Decision (D3) Framework

The authors propose the Decoupled Declarative Decision (D3) framework, which separates what a decision is (the logic) from how it is executed. This framework enables zero-modification generalization across diverse expert-defined decision structures.

Logic Primitive Inventory (LPI): A fixed, finite vocabulary of atomic execution primitives ( $V$ $V$ ) partitioned into three types:
- Deterministic Nodes ( $V_{det}$ ): Resolve via closed-form arithmetic predicates over tabular metrics (e.g., height differences).
- VLM Nodes ( $V_{vlm}$ ): Resolve by querying a Vision-Language Model agent on image modalities (e.g., canopy overlap).
- Exit Nodes ( $V_{exit}$ ): Terminate execution and return a class label.
Neural Rule Transpiler (NRT): An LLM-based component that compiles structured natural language expert rules ( $\rho$ ) into a typed, executable decision graph ( $T$ ) over the LPI. The NRT ensures the output is a valid JSON structure conforming to the LPI schema.
Orchestrator: A rule-agnostic engine that traverses the compiled decision graph. It evaluates deterministic nodes directly and delegates perceptual nodes to the VLM agent. The orchestrator contains no hard-coded expert rules, allowing the system to adapt to new rules simply by re-compiling the configuration.

TreeAgent Implementation

TreeAgent is the instantiation of D3 for forestry bias labeling.

VLM Agent: For nodes requiring visual judgment (e.g., detecting crown overlap or ground outliers), the system employs a Vision-Language Model. To mitigate stochasticity, the agent performs $K=3$ independent inference calls at temperature $\tau=0.2$ and aggregates results via majority voting.
Input Modalities: The VLM receives structured tabular data and two specific visual inputs: (1) a bird's-eye CHM window with crown polygons and stem locations, and (2) a two-panel LiDAR transect cross-section (N-S and E-W) colored by LiDAR class.
Prompt Engineering: The system tests various prompt strategies, including Chain-of-Thought reasoning and expert framing, to optimize the VLM's ability to interpret LiDAR-specific rendering conventions.

Baselines

The study compares TreeAgent against:

Supervised ML Baselines: A suite of tabular models (LightGBM, XGBoost, Random Forest, CatBoost, TabPFN) trained on hand-engineered features summarizing tree height sources and local terrain context.
Point Cloud Models: A PointNet architecture trained end-to-end on raw cropped point clouds.
Human Annotation: The reference standard, noting the time cost and variability.

3. Key Contributions

The D3 Framework: A novel architecture that decouples expert classification logic from execution. It introduces a closed Logic Primitive Inventory and an LLM-based Neural Rule Transpiler that compiles natural language rules into validated, auditable decision graphs.
TreeAgent System: A multi-agent system encoding a forestry expert's seven-class bias diagnostic. It utilizes a majority-vote VLM agent to handle perceptual nodes within the decision graph, effectively bridging symbolic rules and visual perception.
Empirical Evaluation: A rigorous evaluation on expert-labeled trees from two held-out NEON sites (WREF and SRER), demonstrating that TreeAgent significantly outperforms supervised ML baselines while drastically reducing annotation time.

4. Results

Performance Metrics

TreeAgent: Achieved a 67.6% Macro-F1 score on the seven-class bias taxonomy with an average runtime of 0.040 minutes per tree.
Supervised ML Baseline (LightGBM): Achieved 36.2% Macro-F1 despite extensive feature engineering, class imbalance correction, and hyperparameter tuning.
Human Annotation: Requires approximately 5.0 minutes per tree.

Sensitivity and Robustness

Rule Fidelity: The system's accuracy is highly dependent on the fidelity of the encoded rule to the expert's actual annotation procedure. Variations in the "No Difference" predicate (e.g., strict vs. non-strict inequality, different normalizers) caused Macro-F1 scores to fluctuate between 21.9% and 67.6%. The headline result (67.6%) was achieved using a rule that strictly matched the annotation procedure used to generate ground truth.
VLM Performance: The VLM agent's performance varied by prompt version. The "V1" prompt (adding system context) achieved 100% recall for crown overlap detection with 25% specificity, balancing the need to catch real overlaps against false alarms.
Geographic Generalization: The supervised ML baseline performed poorly on the geographic split (training on OSBS, testing on WREF/SRER), dropping to 14.2% Macro-F1 on the SRER site. TreeAgent maintained higher performance, suggesting better generalization when guided by explicit expert rules.

Limitations Observed

VLM Errors: The dominant source of labeling error is the VLM agent, particularly in classes requiring visual judgment. General-purpose VLMs struggle with specialized LiDAR rendering conventions.
Threshold Brittleness: Deterministic rules can be brittle at boundary conditions. Two trees nominally falling under a "No Difference" rule were misclassified because their measurements sat at the edge of the 2% tolerance, highlighting a tension between hard symbolic thresholds and perceptual judgment.

5. Significance and Claims

The paper claims that agentic orchestration of VLMs with expert priors offers a viable path toward scalable, interpretable annotation in domains where ground truth is expensive and expert reasoning is structured but occasionally requires perceptual judgment.

Cost and Efficiency: TreeAgent reproduces expert-defined labeling procedures at a fraction of the cost (0.040 mins vs. 5 mins per tree) while maintaining higher agreement with expert references than conventional supervised ML.
Interpretability and Auditability: Unlike end-to-end models, every decision in TreeAgent is traceable to the specific expert rule node that produced it. This preserves the auditability required for high-stakes scientific workflows.
Generalizability: The D3 framework allows for rule revisions via configuration edits rather than code re-engineering, making it adaptable to evolving scientific standards.
Scope: While demonstrated in forestry, the authors posit that D3 provides a general recipe for scientific labeling workflows where expert logic is structured and evolving. They caution that the system is not a substitute for expert review in high-stakes inventories, as VLM errors on perceptual nodes can propagate to downstream estimates, and the system inherits biases present in the encoded expert rules.