Technical Summary: Fine-Tuned Multi-Agent Framework for Detecting OCEAN in Life Narratives

1. Problem Statement

Accurately assessing personality from text is inherently difficult because traits are latent, context-dependent, and often expressed through subtle linguistic cues rather than explicit keywords. While Large Language Models (LLMs) offer the capacity to process extensive textual contexts, they suffer from latent "personality-like" biases inherited from pretraining. Relying on a single model often leads to inconsistent assessments, hallucinations, and skewed inferences, particularly when analyzing long-form life narratives where evidence is distributed and implicit. Existing approaches, ranging from feature-engineered models to prompt-based LLMs, often struggle with short texts, coarse labeling, or a lack of interpretability and theoretical grounding.

2. Methodology

The authors propose a Fine-Tuned Multi-Agent Framework designed to detect the Big Five personality traits (OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) from life narrative transcripts. The framework operates through a hierarchical structure involving specialized sub-agents and a judge agent.

2.1 Multi-Agent Architecture

The system employs 15 sub-agents (three for each of the five traits: High, Neutral, and Low).

• Sub-Agents: Each sub-agent is conditioned to adopt a specific perspective (High, Neutral, or Low) for a specific trait. They perform binary classification to determine if the narrative exhibits the specific trait level they are trained on. They output a binary decision, supporting textual evidence, and a confidence score.
• Judge Agent: A separate LLM acts as a judge. It aggregates the outputs (decisions, evidence, and confidence scores) from the three sub-agents for a given trait, along with the full transcript and trait-specific facet definitions. The judge performs a comparative assessment to reconcile conflicting signals and generate a final multi-class prediction (High, Neutral, or Low).

2.2 Fine-Tuning and Conditioning

To induce personality-sensitive representations and mitigate pretraining biases, the framework utilizes three key mechanisms:

1. MLM Fine-Tuning with Emotion-Aware Masking: Sub-agents are fine-tuned using a self-supervised Masked Language Modeling (MLM) objective on life narratives. Unlike standard MLM, 30% of tokens are masked: 15% are emotional words (identified via SenticNet) and 15% are non-emotional words. This forces the model to internalize emotional and structural linguistic patterns associated with specific trait levels.
2. LoRA Adaptation: Low-Rank Adaptation (LoRA) is applied to the query, key, value, and output projection layers of the base model (Mistral-7B-Instruct). This allows for efficient fine-tuning with approximately 4.2M trainable parameters per agent, balancing pretrained knowledge with trait-specific adaptation.
3. Psychometric Grounding (IPIP-NEO): Sub-agents are prompted with IPIP-NEO facet keys. High-level agents receive positively keyed facet anchors, Low-level agents receive negatively keyed anchors, and Neutral agents receive blended cues. This provides fine-grained, theory-grounded supervision, ensuring agents focus on behavioral definitions rather than generic cues.

2.3 Data and Implementation

The framework is evaluated on the SPAN (St. Louis Personality and Aging Network) dataset, consisting of 1,535 life narrative interview transcripts. Ground truth continuous scores are discretized into High, Neutral, and Low bins using tercile binning. The system uses Mistral-7B-Instruct for sub-agents and Qwen for the judge model, leveraging the latter's larger context capacity for aggregating distributed evidence.

3. Key Contributions

The paper outlines three primary contributions:

1. A Multi-Agent Framework: The introduction of a structured system where LLM sub-agents are induced into High, Low, and Neutral trait orientations via MLM fine-tuning, role prompts, and IPIP-NEO facet keys.
2. Judge-Mediated Aggregation: A mechanism where a judge model reconciles conflicting trait signals from specialized sub-agents, enabling more accurate classification than single-agent inference.
3. Comprehensive Evaluation: A rigorous assessment including quantitative baselines, ablation studies, semantic alignment analysis, and qualitative case studies to demonstrate how the framework infers personality and mitigates bias.

4. Results and Analysis

4.1 Quantitative Performance

• Baseline Comparison: The full multi-agent framework achieved the best macro-F1 scores across OCEAN traits, improving average performance by approximately 8% over the best-performing single-agent model (Qwen with Chain-of-Thought).
• Model Configuration: The optimal configuration used Mistral-7B for sub-agents (efficient at extracting localized evidence from short, facet-specific prompts) and Qwen as the judge (capable of integrating multiple outputs with full transcripts). Using Mistral as the judge resulted in weaker performance due to context capacity limitations.
• Per-Class Accuracy: The framework performed well on High and Low classes but struggled with Neutral classes (e.g., 20–28% accuracy), reflecting the inherent difficulty of identifying moderate trait expressions.

4.2 Ablation Studies

Removing key components significantly degraded performance:

• No IPIP-NEO Keys: Caused the largest drop (15.63% in average F1), highlighting the critical role of psychometric grounding.
• No MLM Fine-Tuning: Resulted in an 11.30% drop, confirming the value of task-specific adaptation.
• No Sub-Agents: A 7.93% drop confirmed that agent-level specialization improves prediction.

4.3 Qualitative and Semantic Analysis

• Semantic Alignment: The full framework showed the highest directional semantic alignment between sub-agent evidence and IPIP-NEO facet keys. Removing facet keys caused alignment margins to drop or become negative.
• Case Study: In an Openness example, a single-agent baseline incorrectly classified a pragmatic, factual narrative as "High Openness" based on career transitions. The multi-agent framework correctly identified the lack of imaginative depth (via Low/Neutral agents) and the pragmatic nature of the exploration (via the Judge), resulting in a correct "Neutral" classification.

5. Significance and Claims

The paper claims that reliable personality inference from long, complex texts requires both decomposition and psychometric grounding. The proposed framework demonstrates that:

• Multi-Agent Reasoning: Aggregating multiple controlled perspectives is superior to single-agent inference for capturing distributed and subtle trait signals.
• Bias Mitigation: Conditioning agents into specific roles and grounding them in psychometric instruments (IPIP-NEO) mitigates the latent biases and inconsistencies of pre-trained LLMs.
• Interpretability: The hierarchical design provides explainable outputs, where the judge's reasoning is grounded in specific evidence and facet-level definitions, offering a scalable method for text-based personality assessment.

The authors conclude that while the approach improves robustness and interpretability, it remains limited by the difficulty of classifying moderate traits, the reliance on structured clinical transcripts, and increased computational costs compared to single-agent models. The work is intended to support research into narrative patterns and cognitive traits, not to make high-stakes decisions without human oversight.