Comparative Analysis of AI Agent Architectures for Entity Relationship Classification

1. Problem Statement

Entity relationship classification (ERC) is a critical task in information extraction, yet it remains challenging in scenarios characterized by:

• Limited labeled data: Traditional supervised learning requires extensive annotation.
• Complex relational structures: Nuanced semantics in specialized domains (e.g., finance, scientific literature) are difficult for standard models to interpret.
• Domain Adaptability: Pre-trained models (like BERT or RoBERTa) often struggle to flexibly adapt to new relation types or domains without task-specific fine-tuning.
• Static Prompting: Standard few-shot prompting with Large Language Models (LLMs) often lacks mechanisms for interactive correction, targeted specialization, or dynamic context adaptation.

The authors aim to address these limitations by evaluating whether multi-agent architectures can outperform standard prompting and approach the performance of fine-tuned models without requiring additional training data.

2. Methodology

The study evaluates three distinct agent-based architectures designed to enhance LLM reasoning for ERC. The experiments were conducted across three diverse datasets: REFinD (Financial), CORE (Scientific), and SemEval 2010 Task 8 (General Domain). Two model backends were primarily used: Gemini 2.5 Flash/Pro and GPT-4o/4o-mini.

A. Baseline

• Few-Shot Prompting: Standard zero-shot, 3-shot, and 5-shot prompting using GPT-4o and Gemini 2.5 Flash served as the control group. No iterative feedback or agent routing was used.

B. Proposed Architectures

1. Generator-Reflection Architecture (Iterative Self-Correction)

• Mechanism: A two-agent cooperative loop.
  • Generator Agent: Performs initial relation classification based on sentence context.
  • Reflection Agent: Critically evaluates the Generator's output, providing specific critiques and suggestions for correction.
• Process: The system operates in an iterative feedback loop (max 3 cycles). The Generator revises its output based on the Reflection Agent's feedback until a termination condition is met (max iterations or no actionable critique).

2. Hierarchical Multi-Agent Architecture (Task Decomposition)

• Mechanism: A centralized routing system with specialized workers.
  • Orchestrator Agent: Analyzes input sentences and dynamically routes them to the most appropriate Specialist Agent based on the relation type (e.g., "Corporate Structure," "Business Interactions," "Market Context").
  • Specialist Agents: Each agent is constrained to a specific subset of relation labels relevant to its domain.
• Process: The Orchestrator verifies the Specialist's classification. If a misclassification or inconsistency is detected, the Orchestrator provides feedback for re-evaluation. This continues for up to 3 attempts.

3. Dynamic-Example Generator Agent (Adaptive Context Construction)

• Mechanism: A novel approach involving real-time example generation and selection.
  • Example Pool Creation: The agent generates positive and adversarial negative examples tailored to the specific input instance. It simultaneously retrieves semantically similar examples from the training set using FAISS embeddings.
  • Dynamic Selection: An Example Selection Agent evaluates the generated and retrieved pool to select the most relevant, diverse, and challenging examples (10 selected from a pool of 40).
  • Classification: The final classifier uses this dynamically assembled, context-rich prompt to predict the relation label.

3. Key Contributions

1. Systematic Comparative Analysis: This is the first study to comparatively assess three distinct agentic strategies (Reflective, Hierarchical, and Dynamic-Example) specifically for relation classification using LLMs.
2. Novel Dynamic Example Generation: The authors introduce a multi-agent mechanism for per-instance dynamic example generation, utilizing cooperative and adversarial prompting to create tailored few-shot contexts, a departure from static or manually curated examples.
3. Multi-Domain Evaluation: The study validates these architectures across three distinct domains (Financial, Scientific, General) using state-of-the-art models (Gemini 2.5, GPT-4o), providing actionable insights into how agent coordination affects generalization.
4. Performance without Fine-Tuning: The work demonstrates that modular agent systems can achieve performance comparable to fine-tuned models without requiring task-specific training data.

4. Results

The experiments utilized Macro and Micro F1 scores. Key findings include:

• Outperformance of Baselines: All three agent architectures consistently outperformed standard few-shot prompting across all datasets.
• Competitive with Fine-Tuning:
  • CORE Dataset: The Generator-Reflection architecture (Gemini 2.5 Pro + GPT-o3) achieved an F1-Micro of 0.76, surpassing Llama-3 (0.70) and BERT-Prompt (0.74), and approaching the fine-tuned model performance (0.80).
  • SemEval Dataset: The Dynamic-Example agent achieved the highest scores (F1-Micro 0.64), outperforming GPT-4 (0.65) and approaching fine-tuned T5-large (0.80) and Mistral-7B (0.77) relative to their zero-shot baselines.
  • REFinD Dataset: The Dynamic-Example agent yielded the best F1-Micro (0.64), demonstrating superior adaptability in complex financial contexts.
• Architecture Specificity:
  • Generator-Reflection excelled in the scientific domain (CORE), likely due to the need for iterative logical refinement.
  • Dynamic-Example performed best in Financial (REFinD) and General (SemEval) domains, suggesting that adaptive context construction is highly effective for diverse or ambiguous relation types.
  • Hierarchical Multi-Agent showed strong routing accuracy, particularly in the structured REFinD dataset (routing F1 up to 0.98), proving effective for domain-specific task decomposition.

5. Significance and Conclusion

• Practical Guidance: The study provides a roadmap for designing modular, generalizable LLM-based systems. It suggests that for tasks requiring high precision in specialized domains, Dynamic-Example Generation or Hierarchical Specialization are superior to simple prompting.
• Resource Efficiency: These architectures offer a viable alternative to fine-tuning, which is often computationally expensive and data-hungry. They achieve high performance in zero-shot and low-resource settings.
• Future Directions: The authors propose extending this framework to document-level relation extraction, cross-lingual adaptation, and integrating multi-modal signals.

In summary, the paper establishes that agentic coordination—whether through reflection, hierarchy, or dynamic context generation—significantly enhances the reasoning capabilities of LLMs for structured relation extraction, bridging the gap between zero-shot prompting and fine-tuned performance.