📄 agriculture

A Hybrid Data Analytics Framework Integrating Machine Learning and Graph-Based Knowledge Models for Explainable Soil Organic Carbon Assessment

This paper proposes and evaluates a hybrid data analytics framework that integrates Random Forest prediction models with graph-based knowledge systems to enhance the accuracy, spatial contextualization, and interpretability of soil organic carbon assessments in agroforestry systems, as demonstrated through a case study in Mexico.

Original authors: Emmanuel Papadakis, Gulger Mallik, Ahmed Aman Ibrahim, George Baryannis, Andreas Altinalmazis-Kondylis

Published 2026-06-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Emmanuel Papadakis, Gulger Mallik, Ahmed Aman Ibrahim, George Baryannis, Andreas Altinalmazis-Kondylis

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why Do We Need This?

Imagine you are trying to figure out how much "fertilizer" (organic carbon) is hidden inside a giant, messy garden (the soil). This is important because healthy soil acts like a sponge that soaks up carbon from the air, helping fight climate change.

However, checking the soil is tricky. The data we have is like a pile of puzzle pieces from different boxes: some pieces are missing, some are measured in inches and others in centimeters, and they come from different depths. Also, the computers we usually use to guess the carbon levels are like "black boxes"—they give you a number, but they can't tell you why they got that number.

The authors of this paper built a hybrid framework (a mix-and-match toolkit) to solve this. They combined two types of smart tools:

Machine Learning: A super-fast calculator that guesses the numbers.
Graph-Based Knowledge: A map and a logic puzzle that explains why those numbers make sense.

They tested this toolkit on a specific garden: Mexico.

The Three Parts of the Toolkit

1. The "Translator" (Data Harmonization)

Before the smart tools could work, the messy data had to be cleaned up.

The Problem: One soil sample might be measured from 0 to 10 cm deep, while another is 0 to 20 cm. You can't compare them directly.
The Solution: The authors used a mathematical "translator" (called Equal Area Quadratic Spline) to stretch or shrink all the measurements so they all fit into the same standard box: 0 to 30 cm deep.
The Analogy: Imagine you have a pile of socks of all different sizes. Before you can count them or sort them, you have to fold them all into the exact same size square so they stack neatly. This step made the data stackable.

2. The "Calculator" (Machine Learning)

Once the data was clean, they used a Random Forest model.

What it does: Think of this as a team of 100 different experts (trees) who all vote on the answer. Each expert looks at different clues (like temperature, rain, soil texture, and rock type) to guess how much carbon is in the soil.
The Result: The team did a decent job. They could predict the carbon levels with moderate accuracy (getting about 54% to 78% of the pattern right).
The Catch: Like any human guesser, they weren't perfect. Sometimes they were off by a bit. This is normal because soil is complicated.

3. The "Map-Maker" and "Storyteller" (Graph Models)

This is where the paper gets unique. Instead of just giving a number, they added two layers of context:

The Map-Maker (Graph Neural Network):
- How it works: They treated every soil sample as a dot on a map. If two dots were close to each other (within 10 km), they drew a line between them.
- The Magic: The computer looked at these connections to group soils into "neighborhoods" or archetypes. It found 8 distinct types of soil neighborhoods in Mexico based on their location and texture, without even looking at the carbon numbers first.
- The Analogy: Imagine sorting people into groups based on their neighborhood and hobbies, not their bank account. You might find that people in "Neighborhood A" all like gardening, even before you ask them how much money they have. This helps scientists see if a soil sample fits its neighborhood or if it's an outlier.
The Storyteller (Bayesian Network):
- How it works: This is a logic graph that asks, "If the carbon is high, what else is usually true?"
- The Result: It found patterns like: "High carbon usually happens in shallow, acidic soil with grasslands," while "Low carbon usually happens in deep, alkaline soil with lots of rocks."
- The Analogy: It's like a detective's board. If you see a clue (High Carbon), the board lights up to show you the other clues that usually go with it (Acidic pH, Grasslands). This helps humans understand why the calculator gave that number.

The Dashboard: Putting It All Together

The authors built a prototype dashboard (a digital control panel) to show how this works in real life.

What you see: A map of Mexico.
What you can do: You click on a spot, and it tells you:
1. The Number: "There is X amount of carbon here."
2. The Context: "This spot belongs to 'Soil Neighborhood #2'."
3. The Story: "This makes sense because the soil is shallow and acidic, which usually means high carbon."

What Did They Learn? (The Results)

The Calculator works, but isn't perfect: It captured the general trends but couldn't predict every single spot perfectly. This is expected because soil is messy.
The Map-Maker found hidden patterns: It successfully grouped soils into 8 distinct regional types that made sense geographically.
The Storyteller made it explainable: It successfully linked high carbon to specific conditions (like acidic pH and shallow depth), giving humans a reason to trust the number.

The "But..." (Limitations)

The authors are very honest about what this tool cannot do yet:

It's a prototype, not a final product: It's a research tool, not a system ready to issue official carbon credits for money yet.
It needs more data: The data they had for Mexico was a bit small and missing some details (like exactly how farmers manage the land).
It's specific to Mexico: You can't just take this model and run it in Brazil or Kenya without checking it first.
No "Magic" Causality: The "Storyteller" shows what usually happens together, but it doesn't prove that one thing caused the other. It's a strong hint, not a scientific law.

Summary

This paper is about building a smarter, more transparent way to guess soil carbon. Instead of just giving a number from a "black box," they built a system that gives you the number, shows you where it fits on the map, and tells you the story of why that number makes sense. It's a step toward making climate science more trustworthy and understandable for everyone.

Technical Summary: A Hybrid Data Analytics Framework Integrating Machine Learning and Graph-Based Knowledge Models for Explainable Soil Organic Carbon Assessment

Problem Statement
Reliable assessment of Soil Organic Carbon (SOC) is critical for carbon removal initiatives, agroforestry planning, and credible Measurement, Reporting, and Verification (MRV) systems. However, current approaches face significant hurdles, including heterogeneous soil data, inconsistent sampling depths, limited spatial coverage, and the "black box" nature of data-driven prediction models. Purely machine learning-based models often struggle with interpretability and fail to account for spatial dependencies, treating related observations as independent. This lack of transparency undermines trust in carbon markets, where stakeholders require not only accurate predictions but also auditable explanations of the environmental conditions and assumptions underlying those estimates.

Methodology
The paper proposes a hybrid data analytics framework designed to integrate supervised machine learning with graph-based knowledge models to enable explainable SOC assessment. The framework operates through four modular components:

Data Harmonisation Pipeline: The system ingests heterogeneous data from the World Soil Information Service (WoSIS) and enriches it with climate (World Bank CCKP), land-cover (MODIS), ecoregion (CONABIO), and expert-recommended soil covariates. A critical step involves standardizing soil measurements to a common 0–30 cm depth interval using the Equal Area Quadratic Spline method (via the ea_spline function in R) to ensure comparability across different sampling protocols.
Supervised Machine Learning Component: This component serves as the primary predictive engine. It employs Random Forest regression models to estimate two key targets: organic carbon concentration (g/kg) and carbon stock (t/ha). The models are trained on harmonized soil and environmental covariates, with strict protocols to prevent data leakage (e.g., profile-level train-test splitting). An exploratory deep learning model was also tested but yielded lower performance than Random Forest.
Graph-Based Knowledge Component (Spatial Representation): To address spatial context, the framework constructs a spatial graph where soil profiles are nodes connected by geographic proximity (edges defined by a 10 km Haversine distance threshold). A Graph Convolutional Network (GCN) is trained using a self-supervised triplet margin loss to learn low-dimensional embeddings of soil profiles. Crucially, organic and inorganic carbon variables are excluded from the input features, ensuring the resulting clusters represent "soil archetypes" based on environmental and pedological similarity rather than direct carbon values.
Explanatory Graph Component: A Bayesian-network-based graph is constructed to support probabilistic reasoning. This component models assumed relationships between SOC and variables such as pH, depth, slope, rock fragments, land cover, and ecoregion. By conditioning the graph on high or low SOC states, the system generates interpretable environmental profiles that explain the conditions associated with different carbon levels.

Key Contributions
The paper outlines six primary contributions:

Hybrid Framework: Proposes a unified workflow combining supervised prediction, graph-neural-network-based spatial representation, and probabilistic graph-based explanation.
Data Harmonisation: Develops a workflow that integrates disparate soil and environmental datasets, standardizing measurements to a 0–30 cm depth interval.
Predictive Evaluation: Evaluates supervised models for estimating SOC concentration and stock in a Mexico-focused case study.
Spatial Archetypes: Constructs a GCN embedding model that identifies spatially coherent soil archetypes independently of carbon values.
Probabilistic Explanation: Demonstrates how graph-based reasoning can interpret environmental profiles linked to high and low organic carbon conditions.
Prototype Implementation: Delivers a Streamlit-based dashboard that visualizes predictions alongside spatial clusters and explanatory profiles, serving as a proof-of-concept for decision support.

Results
The framework was evaluated using a Mexico-focused case study with WoSIS data.

Predictive Performance: The Random Forest models achieved moderate performance, with $R^2$ values ranging from 0.54 to 0.78. The Total Carbon Equivalent (TCEQ) concentration model performed best ( $R^2$ = 0.78), while stock predictions showed $R^2$ values of 0.54. The authors note that while the models capture broad spatial trends, they exhibit considerable uncertainty for precise point-level estimation. The exploratory deep learning model underperformed ( $R^2$ = 0.26) and was not extended to stock prediction tasks.
Spatial Clustering: The GCN embedding model successfully identified eight spatially coherent soil clusters across Mexico. These clusters corresponded to interpretable differences in soil texture, pH, and slope, validating the model's ability to capture regional soil similarities without direct reliance on carbon data.
Explanatory Profiling: The Bayesian network successfully differentiated environmental profiles for high versus low SOC. High SOC was associated with acidic pH, shallow soil layers, and temperate sub-humid ecoregions. Conversely, low SOC was linked to alkaline conditions, deeper layers, arid regions, and higher rock-fragment content.

Significance and Claims
The authors position this work as a step toward more transparent and interpretable soil carbon assessment, rather than a fully operational MRV platform. The significance lies in the integration of three complementary evidence types: numerical prediction, spatial similarity, and probabilistic explanation.

The paper claims that this hybrid approach addresses the "explainability gap" in environmental AI. By linking a numerical carbon estimate to a spatially grounded soil archetype and a probabilistic environmental profile, the framework allows users to understand why a prediction was made and whether it aligns with known regional patterns. This is presented as essential for building trust in carbon markets and supporting expert review in MRV workflows.

However, the authors maintain a modest stance regarding operational readiness. They explicitly acknowledge limitations, including the relatively modest size of the harmonized dataset, the lack of detailed land-use/management data (which limits the assessment of agroforestry impacts), and the need for further validation before the framework can support high-stakes carbon crediting. The current output is described as a research prototype designed to demonstrate analytical feasibility and the value of hybrid analytics, not as a final solution for carbon-credit verification.