🤖 machine learning

LLM for the development of FCM

This paper demonstrates that a local large language model (Qwen2.5-32B) can extract quantitative data from unfiltered TripAdvisor hotel reviews to construct and validate a data-driven Fuzzy Cognitive Map that effectively correlates reviewer preferences with star ratings.

Original authors: Alexis Kafantaris

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexis Kafantaris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, chatty robot brain living on your own computer. This robot, named Qwen2.5-32B, is like a super-smart detective that can read thousands of hotel reviews and figure out exactly what makes a traveler happy. The paper by Alexis Kafantaris shows us how this local robot detective can build a "Fuzzy Cognitive Map" (FCM), which is basically a digital spiderweb of cause-and-effect that explains why people give hotels good or bad stars.

The Detective's Mission: Turning Words into Numbers

Usually, to understand hotel reviews, you might try to count words or guess the mood. But this paper suggests a smarter way: let the robot read the text and pull out specific numbers, like a score for "Staff," "Breakfast," or "Noise."

Think of it like a chef tasting a soup. Instead of just saying "it's good," the robot tastes it and says, "The salt is +0.6, the pepper is -0.1, and the broth is +0.7." The robot did this for 1,505 Greek hotel reviews from TripAdvisor. It turned messy, chatty text into a neat table of numbers.

Building the Spiderweb (The FCM)

Once the robot had the numbers, the researchers built the Fuzzy Cognitive Map. Imagine a spiderweb where the center is the "Satisfaction" of the guest. Tied to this center are nine different strands, each representing a hotel feature like Staff, Value, Comfort, or Location.

The researchers taught the spiderweb how strong each strand is using a mathematical trick called gradient descent. It's like a hiker trying to find the bottom of a valley by taking small steps downhill; the hiker keeps adjusting the path until they find the perfect spot where the error is smallest. In this case, the "hiker" adjusted the strength of the connections between the features (like Staff) and the final feeling (Satisfaction) until the model could accurately predict how happy a guest would be.

What the Spiderweb Revealed

The results were surprisingly clear. The robot found that for Greek reviewers, the two strongest strands holding up the "Satisfaction" center were Staff and Value.

Staff had a weight of +0.65.
Value had a weight of +0.56.
Comfort and Amenities were also important, but Check-in and Cleanliness were surprisingly weak links, contributing very little once the other factors were considered.

The paper explicitly rules out the idea that smaller robot brains (like the 0.5B or 7B versions) could do this job. The authors found that smaller models were "hallucinating" (making things up) or getting the signs wrong (saying something was good when it was bad). Only the bigger, smarter Qwen2.5-32B model could extract the data correctly. They also tried a tool called VADER (a standard sentiment analyzer), but it failed miserably because it couldn't handle mixed feelings or specific categories in hotel reviews.

Is the Spiderweb Real? (The Proof)

How do we know this isn't just a lucky guess? The researchers ran a series of tests:

The "What If" Test: They shuffled the answers around randomly 200 times. When they did this, the model failed completely (getting a score of -0.324). This proves the model learned real patterns, not just noise.
The "Star Rating" Test: Here is the coolest part. The robot never saw the star ratings (1 to 5 stars) during its training. It only saw the text and the extracted numbers. Later, the researchers compared the robot's predicted "Satisfaction" score against the actual star ratings the hotels received.
- For hotels with at least 10 reviews, the robot's prediction matched the star rating with a correlation of +0.807.
- This suggests that if the robot thinks a guest is satisfied based on the text, that guest is very likely to give the hotel a high star rating.

The Limits of the Magic

The paper is careful not to call this a perfect, solved problem.

It's not the strongest predictor: If you just want the highest possible score for predicting a number, other tools like XGBoost or Random Forest actually did better (scoring +0.856 vs. the FCM's +0.782).
The Trade-off: The reason to use this FCM spiderweb isn't raw power; it's clarity. You can look at the web and see exactly why the robot thinks a hotel is good (because of the Staff strand). You can't see inside the "black box" of the other tools.
The Cost: Running this local robot brain is expensive and slow. It requires a powerful computer (GPU) and takes hours to process even a few thousand reviews. The authors suggest that while it works, it's not yet a cheap, instant solution for millions of reviews.

The Bottom Line

This paper suggests that a local AI robot can read hotel reviews, turn them into numbers, and build a clear, understandable map of what makes guests happy. It proves that Staff and Value are the kings of Greek hotel satisfaction. While it's not the fastest or most powerful tool in the shed, it offers a unique, transparent view into human feelings that other tools can't provide. The authors conclude that this method works well enough to be useful, but future work needs to find cheaper ways to run it and maybe add more features to the map.

Technical Summary: LLM for the Development of FCM

Problem Statement
The paper addresses a gap in the application of Large Language Models (LLMs) to the construction of Fuzzy Cognitive Maps (FCMs). While previous research has utilized LLMs to directly extract causal relationships between concepts (e.g., Kosko's approach), this work proposes a distinct pipeline: using a local LLM to extract quantitative data (sentiment scores) from textual inputs, which are then used to train a data-driven FCM via gradient descent. The authors note that prior work relying on LLMs for direct causality extraction often requires expensive, state-of-the-art models, and that the potential of using LLMs merely as data extraction engines for subsequent FCM training has not been thoroughly explored.

Methodology
The proposed framework follows a five-step pipeline to transform unstructured hotel reviews into a predictive, interpretable model:

Data Sourcing and Preprocessing: The study utilizes 1,505 Greek hotel reviews from TripAdvisor.
Aspect-Based Sentiment Analysis (ACSA) via Local LLM: Instead of traditional sentiment analysis tools (which the authors found limited, e.g., VADER failed at mixed sentiment extraction), the authors employ a local LLM, specifically Qwen2.5-32B. This model performs zero-shot prediction to assign continuous sentiment scores to a fixed set of nine concept categories derived from the Hontology (Cleanliness, Staff, Location, Breakfast, Noise, Value, Comfort, Amenities, Check-in). The authors determined that smaller models (e.g., Qwen2.5-0.5B, Mistral-7B) produced noisy signals or hallucinations, while the 32B parameter model provided valid, high-accuracy extraction.
Data-Driven FCM Construction: The extracted sentiment scores serve as input features for a Fuzzy Cognitive Map. The FCM is modeled as a shallow recurrent neural network where concepts are nodes and causal relationships are weighted edges.
Training: The weights (specifically the driver-to-satisfaction connections) are learned using gradient descent to minimize the squared prediction error between the model's output and the target satisfaction score. The activation function used is tanh.
Evaluation: The system is evaluated using a 70/30 held-out split, k-fold cross-validation (5-fold), ablation studies, and permutation null tests.

Key Results

Predictive Performance: The data-driven FCM achieved a held-out $R^2$ of 0.795, significantly outperforming a linear regression baseline (0.669) and a mean estimator (-0.000). While it did not surpass the raw predictive power of black-box models like XGBoost (0.856) or Random Forest (0.850), it maintained high performance while offering interpretability.
Ablation and Feature Importance: The ablation study revealed that Staff and Value are the most critical drivers for satisfaction. Removing these concepts caused the largest degradation in model performance. Conversely, "Check-in" and "Cleanliness" contributed negligibly once other drivers were present.
Stability: The model demonstrated stability across 5-fold cross-validation, with low standard deviations in the learned weights (e.g., Staff weight: $0.649 \pm 0.013$ ).
External Validation: A crucial validation step involved correlating the model's predicted satisfaction scores with the actual star ratings of the hotels on TripAdvisor. The model had never seen star ratings during training. A strong positive Pearson correlation was observed, reaching 0.807 for hotels with at least 10 reviews, confirming that the extracted sentiment signals align with real-world user ratings.

Key Contributions

Novel Pipeline: The paper introduces a specific workflow where a local LLM acts as a data extraction engine to generate tabular sentiment data, which is then used to train a gradient-descent-based FCM, rather than using the LLM to infer causality directly.
Local LLM Viability: It demonstrates that local models (Qwen2.5-32B) can effectively perform complex aspect-based sentiment analysis on domain-specific text (hotel reviews) with sufficient accuracy to train downstream models, offering a privacy-preserving alternative to cloud-based SOTA models.
Interpretability: The resulting FCM provides a "star topology" map where the causal influence of specific attributes (e.g., Staff, Value) on overall satisfaction is quantified and visualized, offering insights that black-box models cannot provide.

Significance and Claims
The authors claim that this implementation successfully bridges the gap between unstructured text and structured system dynamics modeling. The significance lies in the ability to create a data-driven, interpretable FCM without relying on manual expert elicitation for weight definition or expensive LLMs for direct causal inference. The strong correlation between the model's internal "satisfaction" metric and external TripAdvisor star ratings validates the pipeline's ability to capture genuine user preferences.

The paper maintains a modest tone regarding limitations, acknowledging that while the local LLM approach is effective, it is computationally expensive (requiring GPUs) and limited in scale compared to cloud APIs. Furthermore, the authors note that while the FCM is interpretable, it is not the absolute best predictor in terms of raw accuracy compared to ensemble methods like XGBoost; its primary value is in understanding the mechanism of user satisfaction rather than maximizing predictive power alone. Future work is suggested to expand the entity set and utilize the FCM as a personalized recommendation tool.