💬 NLP

Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System

This paper proposes a comprehensive, multi-factor scoring system with a graphical interface to evaluate large language models across dimensions like accuracy and coherence, revealing their reasoning strengths and factual limitations on the TruthfulQA dataset while offering a transparent framework for future model refinement.

Original authors: Yiming Gai, Junde Lu, Xuefei Huang

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiming Gai, Junde Lu, Xuefei Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to answer questions for your company. In the past, you might have just asked, "Did they get the facts right?" and stopped there. But what if they got the facts right but wrote a 50-page novel to say it? Or what if they were right but sounded like a robot from the 1950s?

This paper introduces a new way to grade these AI assistants (called Large Language Models or LLMs) that is much more like a comprehensive report card rather than just a simple pass/fail test.

Here is the breakdown of their new system, using simple analogies:

1. The Problem: The "One-Dimension" Trap

Think of old evaluation methods like BLEU or ROUGE as a teacher who only checks if a student used the exact same words as the answer key. If the student wrote a brilliant explanation using different words, the old teacher might give them a bad grade. The authors argue this is too narrow. It's like judging a chef only on whether they used the exact same ingredients as the recipe, ignoring whether the food actually tasted good or was easy to eat.

2. The Solution: The "Six-Point" Scorecard

The authors built a new system that grades AI responses on six different pillars, much like a car safety rating that checks brakes, airbags, fuel efficiency, and comfort, not just speed.

Here are the six factors they measure:

Accuracy (The Truth): Did the AI get the meaning right? They use a "semantic compass" (math that measures how close the ideas are, not just the words) to see if the answer matches the truth.
Conciseness (The Brevity): Is the AI rambling? If the answer is twice as long as it needs to be, it gets a penalty. It's like a friend who tells a 10-minute story when you just asked for the time.
Factual Consistency (The Fact-Checker): Does the AI include the specific key facts from the real answer? They check if the "ingredients" (key words) in the AI's bowl match the recipe.
Readability (The Flow): Is it easy to read? They check if sentences are too long or if the vocabulary is too repetitive. It's like checking if a book is written in plain English or in confusing jargon.
Coherence (The Logic): Do the sentences stick together? They check if sentence A leads logically to sentence B, ensuring the story doesn't jump around randomly.
ROUGE Score (The Overlap): This is the "old school" check to see how many words match the reference answer, just to be safe.

3. The Test Drive: The "TruthfulQA" Track

To test this new scoring system, the authors took five popular AI models (including models from Alibaba, ByteDance, Google, and others) and put them through a specific obstacle course called TruthfulQA.

Think of this course as a "trap-filled maze" designed to trick AI. The questions aren't just "What is 2+2?"; they are tricky things like, "Is the moon made of green cheese?" or questions based on common myths. The goal was to see which AI could navigate the traps without falling for the lies.

4. The Results: Who Won the Race?

The study found that no single AI was perfect at everything. It's like a sports team where one player is great at defense but slow at running, while another is a speedster but bad at defense.

Gemini 2.0 Flash came out on top with the highest overall score (0.6104). It was the most balanced, especially great at accuracy and keeping things short.
DeepSeek-v3 was the "Readability Champion." Its answers were the easiest to read and flowed the best.
Doubao-1.5-pro-32k was the "Fact-Checker," scoring highest on sticking to the specific facts.
Moonshot-v1-8k struggled the most, particularly with confusing questions about people.

The Big Discovery:
All the models were surprisingly good at logic puzzles (like spotting a logical lie), but they all hit a wall when faced with complex misinformation or confusing real-world facts. They tended to get tripped up by the "traps" in the maze.

5. The Takeaway

The authors built a Graphical User Interface (GUI), which is basically a dashboard that lets you see these scores visually, like a radar chart showing a model's strengths and weaknesses.

In summary: This paper doesn't just ask, "Is the AI smart?" It asks, "Is it smart, concise, truthful, easy to read, and logical?" By using this multi-factor scorecard, we can finally pick the right AI tool for the right job, rather than just guessing which one is "best." The study focused entirely on English text, but the authors suggest this method could eventually be used for other languages too.

Technical Summary: Comprehensive Evaluation of Large Language Model Responses

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable performance in linguistic tasks, existing evaluation methods suffer from significant limitations. Traditional approaches often rely on single-dimensional metrics (e.g., accuracy or fluency) or surface-level n-gram matching (e.g., BLEU, ROUGE), which fail to capture the full spectrum of model capabilities. Specifically, current methods frequently overlook user-experience dimensions such as readability and conciseness, and struggle to provide a holistic assessment of factual consistency and logical coherence in open-domain question answering. This gap is particularly critical in specialized domains like healthcare and law, where diverse practical needs demand a more nuanced evaluation framework.

Methodology

The paper proposes a Multi-Factor Scoring System (MFSS) designed to quantitatively assess LLM response quality across six distinct dimensions. The system employs a modular architecture consisting of data preprocessing, semantic embedding, and multi-factor evaluation modules.

Core Evaluation Dimensions

The framework integrates the following metrics, computed using a combination of semantic embedding (via the all-MiniLM-L6-v2 model) and traditional text analysis:

Accuracy: Measures semantic similarity between the model response ( $A_m$ ) and the reference answer ( $A_g$ ) using cosine similarity of embedding vectors, prioritizing semantic alignment over exact word matches.
Conciseness: Evaluates succinctness by calculating an inverse redundancy metric. It penalizes repetitive sentences and responses that exceed the length of the reference answer by a factor of two, using a combination of exponential and quadratic penalties.
Factual Consistency: Assesses the alignment of key factual elements using a set-overlap approach. It tokenizes text, removes stopwords, and calculates the intersection of word sets between the model response and the ground truth.
Readability: Based on linguistic theory, this metric combines a Sentence Length Score (deviation from an optimal 17.5 words) and Lexical Diversity (proportion of unique words).
Coherence: Evaluates logical flow by computing the average cosine similarity between embedding vectors of adjacent sentences within the response.
ROUGE Score: Complements semantic evaluation by quantifying n-gram overlap. It integrates ROUGE-1, ROUGE-2, and ROUGE-L F1 scores with specific weighting.

Scoring Mechanism

A Total Score (TS) is derived via a weighted average of the six metrics. The default weights are empirically determined as follows:

Accuracy: 0.25
Factual Consistency: 0.20
Readability: 0.15
Coherence: 0.15
ROUGE Score: 0.15
Conciseness: 0.10

The system includes a Graphical User Interface (GUI) to visualize outcomes and supports dynamic weight adjustments for different scenarios.

Experimental Results

The framework was validated using the TruthfulQA dataset, comprising 817 English questions across categories such as science, history, and health. Five mainstream LLMs were evaluated: qwq_plus_latest, deepseek_v3, doubao-1-5-pro-32k, moonshot-v1-8k, and gemini-2.0-flash.

Key Findings:

Overall Performance: gemini-2.0-flash achieved the highest composite score (0.6104), followed by deepseek_v3 (0.6088) and doubao-1-5-pro-32k (0.5873).
Strengths: Models generally demonstrated strong capabilities in reasoning tasks, particularly in categories like "Logical Falsehood" and "Mandela Effect," where scores often exceeded 0.70.
Limitations: All evaluated models exhibited pervasive limitations in navigating complex facts and ambiguities, specifically in "Misinformation" categories, where scores dropped significantly (e.g., doubao-1-5-pro-32k scored 0.2497, and moonshot-v1-8k scored 0.2936 in specific confusion categories).
Model Specifics:
- Gemini 2.0 Flash: Excelled in accuracy and conciseness.
- DeepSeek-v3: Showed particular strength in readability and coherence.
- Doubao-1.5-pro-32k: Demonstrated superior factual consistency.

Key Contributions

Multi-Dimensional Framework: The study introduces a novel scoring system that moves beyond single metrics by integrating user-experience dimensions (readability, conciseness, coherence) with traditional accuracy and factual consistency measures.
Enhanced Transparency and Applicability: By combining semantic embeddings with ROUGE metrics and providing a GUI for visualization, the framework offers a scalable solution applicable to both open-domain QA and specialized domains (e.g., law, healthcare).
Empirical Insights: The multi-factor analysis systematically reveals specific model strengths in reasoning versus their weaknesses in handling complex factual scenarios, providing data-driven guidance for model optimization.

Significance and Claims

The paper posits that this multi-factor scoring paradigm offers a transparent and adaptable avenue to illuminate both the potential and deficiencies of LLMs, transcending the narrow lens of traditional metrics. While the current implementation is focused on English tasks using the TruthfulQA dataset, the authors claim the framework carves a novel path for knowledge engineering and model refinement. The study suggests that this approach facilitates more informed model selection and refinement strategies, with future horizons beckoning toward multilingual domains, multimodal applications, and integration with knowledge graphs.