AgriMemSynth: a synthetic benchmark for memory and multi-hop reasoning in agricultural question answering

Technical Summary: AgriMemSynth

Problem Statement
Agricultural advisory systems face significant challenges in knowledge discovery due to the heterogeneous nature of agricultural data, which spans images, sensor readings, textual observations, and expert knowledge. Existing agricultural benchmarks, such as PlantVillage and PlantDoc, primarily focus on image classification or short-form question answering (QA). These repositories fail to capture the conversational context of farmer-expert interactions, the temporal progression of symptoms across multiple visits, or the complex multi-hop reasoning required to connect treatments to specific diseases, crops, and environmental conditions. Furthermore, current evaluation methods often rely on lexical metrics (e.g., F1, Exact Match) that underestimate answer correctness in agricultural domains, where semantically equivalent answers can be lexically diverse (e.g., "copper-based fungicide" vs. "Bordeaux mixture"). There is a lack of controlled benchmarks that evaluate memory-augmented systems on both longitudinal conversational recall and structured multi-hop reasoning within the agricultural domain.

Methodology
The paper introduces AgriMemSynth, a synthetic benchmark framework designed to evaluate agricultural QA under two complementary requirements: multi-session conversational memory and structured multi-hop reasoning. The framework consists of two primary modules:

1. AgriSynth (Data Generation): This module generates synthetic benchmark datasets grounded in real agricultural resources.

   • Sources: It utilizes 54,306 images from PlantVillage (laboratory conditions) and 2,598 images from PlantDoc (field conditions), alongside structured crop-disease-treatment knowledge bases.
   • AgriConvMem: A dataset of 500 multi-session advisory conversations (350 train, 75 validation, 75 test). These simulate farmer-expert interactions spanning 3–5 sessions, covering disease identification, temporal tracking, severity assessment, and treatment planning. A Vision-Language Model (GLM-4.6V) generates detailed textual descriptions from images to ground the conversations.
   • AgriMultiHop: A dataset of 2,000 structured reasoning questions (1,600 train, 200 validation, 200 test). These require chaining knowledge across disease profiles, treatment databases, and crop compatibility tables. Questions are categorized into five reasoning types: counting, negation, implicit, derived, and comparative.

2. AgriMemory (Evaluation): This module benchmarks five distinct memory architectures under a shared answer-generation pipeline using a fixed backbone language model (DeepSeek-V3, 671B parameters) to isolate architectural differences. The five systems are:

   • RAG (Retrieval-Augmented Generation): Uses dense vector embeddings (BGE-M3) to retrieve flat text chunks based on semantic similarity.
   • NMS (Neuroscientific Memory System): Organizes knowledge into four cognitively inspired stores: Working (active context), Episodic (temporal events), Semantic (factual knowledge), and Procedural (diagnostic routines).
   • Hybrid (NMS+RAG): Combines the structured stores of NMS with the vector retrieval of RAG.
   • BM25 with Semantic Reranking: Uses term-frequency retrieval followed by neural cross-encoder reranking.
   • MemoryGraph: Constructs a knowledge graph of entities and relationships, enabling explicit multi-hop traversal.

Evaluation Metrics
The study employs a dual-metric approach to address the lexical-semantic gap:

• Lexical Metrics: F1, Exact Match (EM), and BLEU-1 to measure surface-level text overlap.
• Semantic Metric: An LLM-as-Judge approach to evaluate meaning-level correctness, acknowledging that agricultural answers can be expressed in various lexically different but semantically equivalent ways.
• Statistical Uncertainty: 95% Wilson confidence intervals are calculated for the binary LLM-as-Judge outcomes to distinguish stable benchmark patterns from sampling variability.

Key Results
Evaluation on held-out test splits (146 AgriConvMem questions and 200 AgriMultiHop questions) yielded three primary findings:

1. Lexical-Semantic Gap: Lexical metrics substantially underestimate system performance. Across all systems, LLM-as-Judge scores were 2–3 times higher than F1 scores (e.g., AgriConvMem F1: 0.26–0.29 vs. Semantic: 0.81–0.86). This confirms that agricultural QA requires domain-aware semantic evaluation.
2. Task Dependency of Architecture: No single retrieval strategy dominates across all tasks.
   • Conversational Memory: RAG achieved the highest semantic correctness (85.6%, 95% CI: 79.0–90.4), outperforming graph-based and neuroscientific approaches in this specific context.
   • Multi-Hop Reasoning: MemoryGraph achieved the highest score (97.0%, 95% CI: 93.6–98.6), demonstrating the superiority of explicit entity-relationship traversal for structured reasoning.
   • Hybrid Limitation: The Hybrid system (NMS+RAG) underperformed compared to its individual components, suggesting that naive combination of memory architectures can introduce retrieval interference and context dilution.
3. Difficulty Hierarchy:
   • Conversational Memory: Temporal questions were the most challenging category (51.1–66.0% semantic correctness), significantly harder than disease identification or severity assessment.
   • Multi-Hop Reasoning: Counting questions showed the most variation between systems, while negation questions were solved consistently by all systems (100%).

Significance and Claims
The paper positions AgriMemSynth as a controlled benchmark for studying how agricultural AI systems organize, retrieve, and apply knowledge across longitudinal and relational tasks. Its primary contributions are methodological:

• It is the first framework to combine synthetic data generation with memory-system evaluation specifically for agricultural QA, integrating multi-session conversational memory and multi-hop reasoning.
• It demonstrates that agricultural QA benchmarks must evaluate multiple retrieval paradigms rather than relying on a single architecture, as performance is highly task-dependent.
• It highlights the necessity of explicit temporal information structures in advisory datasets, as unstructured dialogue text fails to support reliable temporal reasoning.
• It establishes that graph-based knowledge organization is superior for multi-hop reasoning, while dense vector retrieval remains effective for conversational recall.

The authors explicitly note that the study does not claim any of the five systems outperforms a "no-memory" baseline (closed-book LLM), nor does it claim the synthetic data perfectly replicates real-world farmer linguistic patterns. Instead, the framework provides a reproducible protocol for comparing memory architectures and identifying specific gaps in current agricultural AI capabilities, particularly regarding temporal reasoning and the integration of heterogeneous knowledge sources.