📊 statistics

Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas

This paper introduces VirtueMap, a framework that evaluates Large Language Models' ethical decision-making patterns through Aristotelian virtue ethics by ranking responses to dilemmas against human-validated ground truths to generate comparative profiles of virtues like wisdom, justice, and courage.

Original authors: Ioannis Tzachristas, John Pavlopoulos

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ioannis Tzachristas, John Pavlopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe the personality of a new friend, but instead of asking them "Are you a good person?" (which is a hard yes/no question), you ask them how they would handle a series of tricky, everyday situations. You aren't looking for the "right" answer; you are looking for how they prioritize their values.

This is exactly what the paper "Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas" does, but for Artificial Intelligence.

Here is the breakdown of their project, VirtueMap, using simple analogies:

1. The Problem: AI is Too Black-and-White

Usually, when we test AI, we ask: "Is this answer right or wrong?"
But real life isn't like that. Sometimes, telling the truth hurts someone's feelings. Sometimes, being fair means being a little strict. Sometimes, you have to choose between being brave or being careful.
The authors argue that AI models make different choices based on different "priorities." One AI might always choose the honest answer, even if it's blunt. Another might choose the polite answer, even if it's slightly vague. Both are "ethical" in their own way, but they have different personalities.

2. The Solution: A "Character Map"

Instead of giving the AI a pass/fail grade, the authors created a VirtueMap. Think of this like a five-star radar chart (a pentagon shape) that measures an AI's "character" based on five ancient Greek virtues:

Practical Wisdom: Knowing the right thing to do in a specific situation (not just following a rule).
Justice: Being fair and giving everyone what they deserve.
Truthfulness: Being honest and not hiding things.
Courage: Doing the right thing even if it's scary or costly.
Temperance: Knowing when to hold back and not overdo it.

3. How They Tested It: The "Ranking Game"

The researchers didn't just ask the AI to pick one answer. They gave it 7 everyday ethical dilemmas (like "You found a spreadsheet error; do you fix it immediately or wait?").
For each dilemma, there were 5 possible responses.

The Old Way: Ask the AI, "Which one is best?"
The VirtueMap Way: Ask the AI to rank all 5 options from "Most Ethical" to "Least Ethical."

By seeing how the AI ranks all the options, they can see its full "personality." Does it hate the "rude but honest" option? Does it love the "cautious but vague" option?

4. The "Ground Truth": Checking with Humans

How do they know which ranking represents "Courage" or "Justice"?
They didn't just guess. They asked over 100 real humans to look at the 5 options and say, "If we wanted to show Courage, which order would these go in?"
They only kept the "rules" for scoring if 95% of the humans agreed. This ensures the map is based on common sense, not just the authors' personal opinions.

5. The Results: What the AI "Looks" Like

They tested 9 different famous AI families (like GPT, Claude, Llama, etc.) multiple times to make sure the results were stable.

The Good News: The AIs were very consistent. If you asked the same AI the same question twice, it gave a very similar ranking (90% consistency).
The Personality Differences:
- All the AIs were very good at Practical Wisdom (they liked balanced, thoughtful answers).
- The biggest differences appeared in Courage, Temperance, and Justice.
- Example: One AI might be very "Courageous" (always picks the direct, risky truth), while another is very "Temperate" (always picks the safe, cautious route). Neither is "broken"; they just have different profiles.

6. The Interactive Website

The authors built a website where you can play the game. You rank the dilemmas, and the site draws your "Virtue Pentagon." It then compares your shape to the shapes of the 9 different AIs to see which AI has a personality most similar to yours.

The Bottom Line

This paper doesn't say AI has a soul or is truly "good" or "bad." Instead, it says: "AI models have different ethical styles, just like people do."

VirtueMap is a tool to measure those styles. It moves us away from asking "Is this AI correct?" and toward asking "What kind of ethical character does this AI display?" This helps us understand why an AI makes the choices it does, rather than just judging the final answer.

Technical Summary: Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas

Problem Statement
Large Language Models (LLMs) frequently generate outputs that prioritize competing ethical considerations, such as truthfulness versus tact, or strict fairness versus contextual flexibility. Traditional evaluation methods often treat these behaviors as binary "right" or "wrong," obscuring the underlying structure of ethical decision-making. The authors argue that a model favoring careful compromise is not necessarily less ethical than one favoring direct disclosure; rather, it may express a distinct ethical profile. The challenge is to move beyond single-label judgments to a descriptive framework that captures these nuanced dispositions without claiming models possess literal moral character.

Methodology: The VirtueMap Framework
The paper introduces VirtueMap, a framework that profiles models (and humans) using an Aristotelian virtue-ethics lens. Instead of selecting a single "correct" response, the system requires participants to rank all five responses to a set of ethical dilemmas from most to least ethically preferable.

Virtues and Dilemmas: The framework profiles five Aristotelian virtues: Practical Wisdom (phronesis), Justice (dikaiosyne), Truthfulness (aletheia), Courage (andreia), and Temperance (sophrosyne). The instrument consists of seven general, non-lethal, non-political, and non-religious dilemmas (e.g., "Spreadsheet Error," "Deadline Exception"), each offering five distinct responses (A–E).
Common-Sense Validation: To establish a scoring key, the authors proposed an ordering of the five responses for each dilemma-virtue pair, representing the order from "most to least expressive" of that virtue. These orderings were validated via online questionnaires with over 100 respondents per ordering. An ordering was retained as "operational ground truth" only if at least 95% of respondents confirmed it (with $N > 100$ ). This ensures the scoring key relies on common-sense recognition of virtue relevance rather than author intuition.
Scoring Mechanism: Rankings are scored against the retained virtue-expression orderings using Normalized Borda alignment.
- Borda weights (5, 4, 3, 2, 1) are assigned to ranks.
- The raw alignment $A(d, v)$ is the dot product of the Borda scores of the observed ranking and the virtue ordering.
- This raw score is normalized to a 0–100 scale, where 55 represents perfect agreement with the virtue ordering and 35 represents complete reversal.
- The final score for a virtue is the mean of these normalized scores across all answered dilemmas.
LLM Evaluation Protocol: The authors evaluated nine LLM families (GPT, Claude, Gemini, Llama, DeepSeek, Mistral, MiniMax, Grok, Qwen) via OpenRouter. To account for stochasticity, the evaluation was sequential: models were run multiple times (up to 10) until the 95% confidence interval half-width for virtue means dropped below 5.0 percentage points or consistency reached 95%. The reported profile is the empirical mean of these repeated runs.

Key Contributions

Ranked Ethical Instrument: A novel tool grounded in Aristotelian virtues that requires full ordering of responses rather than single-choice selection.
Validation Criterion: A common-sense validation mechanism for response orderings based on high-threshold (>95%) human confirmation, separating the question of "what an option expresses" from "what a model prefers."
Explicit Scoring Rule: A mathematically defined method (Normalized Borda alignment) to convert rankings into continuous virtue profiles.
Repeated-Run Profiling Suite: A methodology that treats LLM stochasticity as part of the measurement, reporting empirical means and consistency intervals rather than single-run snapshots.
Interactive Implementation: A browser-based website that computes profiles locally, allowing users to compare their own rankings against measured LLM profiles.

Results
The evaluation produced 44 valid full-questionnaire runs across nine model families.

Consistency: The mean rank consistency across models was 90.3%, indicating high stability in how models rank responses across repeated runs.
Virtue Profiles: The highest mean score across all models was for Practical Wisdom (90.4), followed by Truthfulness (82.3), Justice (80.5), Courage (78.0), and Temperance (76.9).
Variability: The largest cross-model differences were observed in Courage (range 10.95), Temperance (range 10.26), and Justice (range 9.70).
Model Specifics: Qwen scored highest on Practical Wisdom; MiniMax on Justice; Claude on Truthfulness and Courage; and GPT on Temperance.
Visualization: The results are visualized as pentagon plots, highlighting that while models converge on Practical Wisdom (often preferring balanced options), they diverge significantly on dimensions requiring intervention (Courage) or restraint (Temperance).

Significance and Claims
The paper positions VirtueMap as a descriptive tool, not a test of moral worth. Its primary significance lies in shifting ethical evaluation from searching for a single correct label to identifying the "disposition expressed by a pattern of choices."

Interpretability: The framework provides a coordinate system interpretable by humans, allowing for the comparison of models that may avoid harmful answers but differ in their prioritization of values (e.g., prompt disclosure vs. cautious delay).
Limitations: The authors explicitly state that the results do not imply LLMs literally possess virtues. The profiles are specific to the questionnaire's scope (seven dilemmas, five virtues) and should be viewed as observed response patterns under a specific protocol, not as fixed properties of the models. The authors acknowledge potential limitations regarding confirmation bias in the validation stage and the operational subset nature of the selected virtues.

The work contributes to the literature on AI and virtue ethics by making Aristotelian virtues the primary representation space for profiling model behavior, rather than treating them merely as a theoretical lens or a subcategory of broader moral benchmarks.