💬 NLP

HAL: Inducing Human-likeness in LLMs with Alignment

The paper introduces HAL, a framework that aligns language models to human-like conversational traits by deriving an interpretable, data-driven reward from contrastive dialogue data, enabling targeted optimization that improves perceived human-likeness without compromising overall performance.

Original authors: Masum Hasan, Junjie Zhao, Ehsan Hoque

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Masum Hasan, Junjie Zhao, Ehsan Hoque

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to chat like a real human. The problem is, "being human" is a fuzzy concept. It's not a math equation you can solve; it's a feeling. You know it when you hear it, but you can't easily write down a rule for it. Usually, to make AI sound more human, developers just throw more money and computing power at the problem, hoping the model gets lucky.

This paper introduces a new framework called HAL (Human Aligning LLMs) that tries to solve this by turning "feeling human" into a measurable score, like a grade on a report card.

Here is how they did it, broken down into simple steps:

1. The Detective Work: Finding the "Tell"

First, the researchers needed to figure out what actually makes a conversation sound human. They looked at thousands of "Turing Tests" (games where a human judge tries to guess who is a person and who is a robot).

Instead of just asking, "Who is human?", they asked a super-smart AI detective to explain why it made its choice. The detective found patterns. For example:

Humans often make small typos or use lowercase letters.
Humans might be a bit impatient or end a chat quickly.
Humans use casual slang and don't over-explain everything.
Humans sometimes admit they don't know something instead of making things up.

The researchers took hundreds of these clues and boiled them down to a checklist of 16 specific traits (called HL16Q). Think of this as a "Humanness Checklist."

2. The Scorecard: Grading the Conversation

Once they had the checklist, they needed a way to grade a conversation. They trained a simple math model (like a weighted scale) to look at a chat and assign it a single number.

If the chat uses casual language and has a few typos, the score goes up.
If the chat is overly polite, perfect, and robotic, the score goes down.

This number is the HAL Score. It's a single number that says, "How human-like does this conversation feel?"

3. The Training: Teaching the Robot to Aim for the Score

Now, they used this score to train different AI models (ranging from small to very large).

They asked the AI to have a conversation.
They gave it a "reward" if the conversation got a high HAL Score.
They gave it a "penalty" if the score was low.

Over time, the AI learned to tweak its behavior to get a higher score. It started acting less like a perfect encyclopedia and more like a real person chatting over coffee.

4. The Proof: Did it Work?

To see if it actually worked, they held a "Blind Taste Test" (similar to a Chatbot Arena). Real human volunteers chatted with two different AIs side-by-side and had to guess which one was the human.

The Result: The AI trained with HAL was chosen as "human" 61.78% of the time.
The Comparison: It beat its own untrained version and even beat a very popular, high-end commercial AI (GPT-4o-mini), which was only chosen as human about 34% of the time.

Why This Matters

The biggest win here isn't just that the AI sounds better; it's that the process is transparent.

Old Way: "We made the AI bigger, and now it sounds more human." (Mystery box).
HAL Way: "We taught the AI to be brief, use slang, and admit when it's wrong, and that's why it sounds human." (Clear recipe).

Because they know exactly which traits they are rewarding, they can check to make sure the AI isn't doing anything weird or harmful to get that score. They also checked to make sure the AI didn't lose its ability to understand emotions while learning to sound human, and it didn't.

In short: HAL is a tool that takes the vague idea of "being human," turns it into a concrete checklist, and uses that checklist to train AI to chat more naturally, without losing its smarts.

Technical Summary: HAL: Inducing Human-likeness in LLMs with Alignment

Problem Statement
Aligning Large Language Models (LLMs) to qualitative behavioral traits, specifically "human-likeness" in conversation, remains a significant challenge. Unlike task-oriented metrics, human-likeness is difficult to define, measure, and optimize systematically. Current improvements in perceived human-likeness are largely driven by model scale or broad supervised training rather than targeted alignment. Furthermore, human judgments of human-likeness are typically holistic and implicit, lacking the explicit criteria necessary to construct a reward signal for reinforcement learning or preference optimization. Consequently, there is no systematic method to align models toward more human-like conversational behaviors.

Methodology
The paper introduces HAL (Human Aligning LLMs), a data-driven framework that quantifies conversational human-likeness into an interpretable, scalar reward signal. The methodology proceeds in four stages:

Trait Extraction from Contrastive Data:
The authors utilize Turing test dialogue transcripts (specifically from Jones and Bergen, 2025) as contrastive data containing paired human-human and human-AI conversations. An LLM-based judge (initially GPT-5 with high reasoning capabilities) analyzes these pairs to generate structured, Likert-style reasons for distinguishing the human from the AI. These natural language reasons are embedded and clustered to identify recurring cues. Through a summarization process, these are distilled into a compact set of 32 characteristics (HL32Q) that reliably distinguish human from AI dialogue in this context.
Quantification via Logistic Regression:
To convert these qualitative traits into a quantitative signal, the authors train a logistic regression model. An LLM judge (HL32Q Judge) independently scores any given dialogue against the 32 traits on a 1–5 Likert scale, producing a feature vector $A \in \mathbb{R}^{32}$ . The logistic regression model learns weights ( $W$ ) and a bias ( $b$ ) to predict whether a dialogue is human ( $y=1$ ) or AI ( $y=0$ ).
To enhance interpretability, the authors perform feature reduction, selecting the top 16 traits (HL16Q) based on the absolute magnitude of their learned weights. This results in a single scalar Human-Likeness Score:
$\text{HumanLikeness}(A) = \sum_{i=1}^{16} A_i W_i + b$
This score serves as the reward signal for alignment.
Preference Optimization (DPO):
The framework frames alignment as a preference learning problem. For a set of synthesized personas (primarily in a medical communication domain), candidate dialogues are generated by various models. These dialogues are scored using the HL16Q Judge. Pairs of dialogues are constructed where the dialogue with the higher HL16Q score is labeled as "chosen" and the lower as "rejected." These ranked pairs are used to fine-tune models of varying sizes (from 1B to 72B parameters) using Direct Preference Optimization (DPO).
Evaluation:
The authors validate the approach through:
- Out-of-Distribution (OOD) Testing: Evaluating the HL16Q score on a separate cancer communication dataset to ensure generalizability.
- Human Evaluation: A Chatbot Arena-style A/B test where human participants judge which of two models feels more human-like.
- Capability Benchmarks: Assessing performance on emotional intelligence benchmarks (EmoBench, EQBench3) to ensure alignment does not degrade other capabilities.

Key Results

Trait Identification: The pipeline successfully identified 32 distinct traits, reduced to 16 (HL16Q), which effectively distinguish human from AI dialogue. The logistic regression classifier achieved 77.47% accuracy on the Turing test dataset using the full 32 traits and 77.12% with the reduced 16 traits.
Generalizability: On an OOD dataset (cancer communication), human-human conversations received significantly higher HL16Q scores than human-AI conversations (mean difference $\Delta = 3.02$ , $p < 10^{-17}$ ), with an AUC of 0.907, indicating the traits capture transferable cues of human-likeness.
Alignment Performance: In human evaluations, the Qwen2.5-14B model aligned with HAL achieved a 61.78% win rate and an Elo score of 1556.97, outperforming both its base counterpart (53.62%) and the proprietary GPT-4o-mini (34.29%).
Preservation of Capabilities: Alignment with HAL did not lead to a systematic degradation in emotional intelligence benchmarks. In several cases, particularly for smaller and mid-scale models, performance on EQBench3 improved after alignment.
Interpretability: The framework allows for the inspection of specific trait changes during training. For instance, the authors observed that the trait "Keeps replies brief and casual" (Q1) saw the most significant distributional shift, validating that the model is learning specific, interpretable behaviors rather than optimizing an opaque objective.

Significance and Claims
The paper claims that HAL demonstrates a viable pathway for aligning LLMs with "soft," qualitative properties that were previously outside the scope of standard alignment objectives. By deriving explicit, interpretable traits from contrastive data and compressing them into a transparent reward signal, HAL enables:

Measurable Alignment: Converting the subjective concept of "human-likeness" into a quantifiable, scalar reward.
Transparency and Diagnosis: Unlike black-box reward models, the HL16Q score allows researchers to inspect which specific conversational traits are being encouraged and to diagnose unintended effects or reward hacking.
Generalizability of Method: The approach offers a general methodology for inducing other difficult-to-specify traits (e.g., sycophancy, manipulation) by inferring them from contrastive examples, provided they can be made measurable.

The authors note that while the method improves perceived human-likeness, the extracted traits are specific to the conversational norms of the training data (Turing tests) and may require domain adaptation for other contexts (e.g., medical dialogue). They also acknowledge the computational cost of using an LLM judge for reward construction but position HAL as a refinement step or a method for richer reward construction where interpretability is paramount.