Technical Summary: Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts

Problem Statement

Current AI-driven approaches to causal graph generation often prioritize high-level causality (e.g., "HIV leads to AIDS") while struggling to capture nuanced, event-specific causal relationships within discrete narratives, such as historical accounts or literary fiction. Existing methods typically follow a two-stage pipeline (causality detection followed by graph construction) but face limitations in interpretability and accuracy, particularly when handling complex sentence structures or implicit causal links. The core challenge addressed by this work is the generation of connected, fine-grained causal graphs from narrative texts that accurately reflect the chain of events leading to tangible outcomes, bridging the gap between high-level causality and specific narrative details.

Methodology

The authors propose an end-to-end framework that integrates linguistic feature extraction with modern language models to generate causal graphs. The process consists of four distinct stages:

1. Vertices Extraction

The system first transforms raw narrative text into a set of concise, agent-centered sentences that serve as graph vertices. Using an LLM-based summarization strategy (specifically GPT-4o, selected for its superior performance in event compression), the text is processed through a structured prompting workflow:

• Summarization: Generating brief summaries of paragraphs.
• Pronoun Substitution: Replacing pronouns with explicit identifiers (e.g., "The Protagonist").
• Clause Simplification: Splitting complex sentences into single-action statements.
• Flow Verification: Ensuring logical continuity while discarding tangential details.
  Each vertex is constrained to contain no more than two clauses, feature a single explicit agent, and utilize active voice.

2. Expert Index Extraction

To enhance classification accuracy, the framework extracts seven linguistically grounded features (the "Expert Index") for each sentence. These features are adapted from discourse frameworks (Hemmatian et al., 2021; Hemmatian Borujeni, 2022) and include:

1. Genericity: Specific vs. generic subjects.
2. Eventivity: Dynamic (action) vs. stative (state) verbs.
3. Boundedness: Episodic, habitual, or static events.
4. Initiativity: Whether the subject initiates or receives the action.
5. Time Start: Past or present relative to the narrative.
6. Time End: Present or future relative to the narrative.
7. Impact: Whether the event's effect persists or is resolved.

A RoBERTa-based classifier is trained on human-annotated data to predict these features for every sentence.

3. STAC Categorization

The system employs a STAC classification model to categorize sentences into four components: Situation (background), Task (requirement), Action (activity), and Consequence (outcome).

• Input: The model combines RoBERTa sentence embeddings (768-dimensional) with one-hot encoded Expert Index features (13-dimensional).
• Architecture: An XGBoost classifier is used to predict the STAC label, leveraging the hybrid input of semantic embeddings and explicit linguistic features.
• Performance: This hybrid approach significantly outperforms models using only embeddings or only linguistic features, as well as zero-shot LLM baselines.

4. Graph Construction

A structured, five-iteration prompting process using LangChain constructs the final causal graph:

1. STAC Bond Learning: The LLM internalizes valid causal connections between STAC categories.
2. Causal Relation Identification: The model evaluates pairs of vertices to propose edges based on STAC bonds.
3. Logical Consistency and Pruning: Counterfactual reasoning ("If A did not occur, would B still happen?") filters out non-causal links.
4. Isolated Vertices Refinement: The model revisits disconnected vertices to identify overlooked causes or effects.
5. Final Graph Construction: The refined vertices and edges are compiled into a coherent, logically sound causal diagram.

Key Contributions

1. Open-Source End-to-End Framework: The authors developed a novel pipeline that significantly improves the interpretability and accuracy of causal graph generation from narrative texts compared to existing methods.
2. Linguistic Feature System: The introduction of the "Expert Index" (seven linguistic features) provides a mechanism to efficiently classify sentences for causal construction. This system is validated through experiments showing that combining these features with semantic embeddings yields superior classification performance.
3. Hybrid Architecture: The paper demonstrates that a hybrid system (RoBERTa embeddings + Expert Index features processed by XGBoost) achieves higher precision in identifying causal links and classifying narrative components than LLM-only baselines, while maintaining lower computational costs.

Experimental Results

The framework was evaluated on a corpus of 100 chapters and short stories (ranging from 1800–1950), including works by O. Henry, Poe, and Fitzgerald.

• Vertices Extraction: GPT-4o was identified as the optimal summarization model, scoring highest in conciseness, coverage, and information economy.
• STAC Classification: The XGBoost model using both RoBERTa and Expert Index features achieved the highest macro-F1 score across all STAC categories. It outperformed the RoBERTa-only baseline by approximately 30% and the GPT-4 zero-shot classifier by 10–15% relative improvement.
• Graph Quality: In pairwise comparisons against GPT-4o and Claude 3.5, the proposed method demonstrated superior performance across most qualitative dimensions:
  • Causality vs. Chronology: 100% win rate against both baselines.
  • Logical Completeness & Accuracy of Connections: 100% win rate.
  • Explicit Motivations/Intent: 95% (vs. GPT-4o) and 92% (vs. Claude 3.5).
  • Granularity & Hierarchy: High win rates (84–94%).
  • Ease of Reading: The proposed method maintained comparable readability to the baselines (52–57% win rate), indicating it does not sacrifice clarity for accuracy.

Significance and Claims

The paper claims that integrating interpretable linguistic feature engineering with modern language models offers a robust solution for fine-grained causal reasoning in narrative texts. By moving beyond pure LLM prompting to a structured pipeline involving linguistic indices and iterative refinement, the framework achieves a level of precision and logical consistency that current state-of-the-art LLMs struggle to replicate autonomously. The authors position this work as a step toward scalable, interpretable analysis of large document collections, enabling stakeholders to trace event chains in social movements, policy-making, and historical trends with greater accuracy. The resulting tool is open-source, aiming to facilitate further research in summarization, discourse analysis, and knowledge graph construction.