💬 NLP

POLAR:A Per-User Association Test in Embedding Space

POLAR is a novel per-user lexical association test that operates in embedding space to detect author-level biases and behavioral shifts, effectively distinguishing LLM-driven bots from organic accounts and quantifying extremist alignment in online communities.

Original authors: Pedro Bento, Arthur Buzelin, Arthur Chagas, Yan Aquino, Victoria Estanislau, Samira Malaquias, Pedro Robles Dutenhefner, Gisele L. Pappa, Virgilio Almeida, Wagner MeiraJr

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pedro Bento, Arthur Buzelin, Arthur Chagas, Yan Aquino, Victoria Estanislau, Samira Malaquias, Pedro Robles Dutenhefner, Gisele L. Pappa, Virgilio Almeida, Wagner MeiraJr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a massive, noisy party where thousands of people are chatting. Some are real humans, some are sophisticated robots, and some are people with very strong, specific opinions.

Traditionally, if you wanted to understand the "vibe" of this party, you would take a microphone, record everyone's voices, mix them all into one giant smoothie, and then taste it. You might say, "This smoothie tastes a bit robotic," or "It has a hint of anger." But you lose the individual flavors. You can't tell who is being robotic or which specific person is getting angrier over time.

POLAR is a new tool that changes the game. Instead of mixing everyone into one smoothie, it gives every single guest a personalized, invisible ID badge that captures their unique "linguistic fingerprint."

Here is how it works, broken down into simple steps:

1. The Invisible ID Badge (The "User Token")

Imagine every person at the party gets a secret, unchangeable name tag (like usr3af12b9cde) that only the computer can see.

The researchers teach a smart computer (a language model) to pay special attention to these name tags.
As the computer reads what a person writes, it learns to summarize their entire personality, writing style, and habits into a single, compact mathematical point in a giant 3D space.
Think of this point as a "GPS coordinate" for that person's mind.

2. The Semantic Compass (The "Axes")

Now, imagine we draw invisible lines (axes) across this 3D space to measure specific things.

Axis A: "Robot vs. Human" (Does this person sound like an AI or a real person?)
Axis B: "Polite vs. Rude"
Axis C: "Immigration Rights vs. Strict Borders"

These lines are made of lists of words. For example, the "Rude" side might have words like slurs or insults, while the "Polite" side has words like "please" and "thank you."

3. The Projection (The "Test")

POLAR takes that person's GPS coordinate (their user vector) and projects a shadow onto these compass lines.

The Result: It tells you exactly where that person lands on the line.
- Do they fall slightly toward "Rude"?
- Do they fall way over to the "Robot" side?
Crucially, it doesn't just guess; it runs a statistical "coin flip" test thousands of times to make sure the result isn't just luck. It gives you a score with a confidence rating.

Why is this a big deal? (The "Aha!" Moments)

The paper tested this tool in two very different scenarios:

Scenario A: The Robot Party (Twitter)
The researchers used POLAR on a mix of real humans and AI bots.

The Old Way: You'd have to train a complex classifier to guess who is a bot.
The POLAR Way: It looked at the "Robot vs. Human" line and instantly saw that the bots were clustered tightly on one side, and humans on the other. It didn't need to be told who was who; the math just showed the difference clearly. It was like seeing a group of people wearing identical, shiny masks vs. people with unique faces.

Scenario B: The Angry Room (Extremist Forum)
They looked at a forum known for hate speech.

The Discovery: POLAR didn't just say "this group is mean." It showed that individual users were drifting.
The Drift: Imagine a user starts posting. At first, they are in the middle of the room. But as they keep posting, POLAR tracks their GPS coordinate moving steadily toward the "Slurs and Hate" side of the compass.
The Insight: This reveals radicalization in real-time. You can see a person slowly turning from a neutral participant into someone deeply aligned with hate speech, just by watching their mathematical coordinate move across the room.

The Bottom Line

POLAR is like a magnifying glass for individual voices in a crowd.

Instead of saying, "The average person here is angry," it allows us to say, "Person X is 80% aligned with hate speech, and they are getting angrier every day," while "Person Y is neutral."

It turns the messy, invisible world of online language into a clear map where we can see individual behaviors, detect bots, and spot dangerous trends before they become a crisis—all without needing to read every single post manually.

1. Problem Statement

Current intrinsic association probes (e.g., WEAT, SEAT) used to detect bias, stereotypes, or semantic associations in language models operate at the word, sentence, or corpus level. These aggregate methods suffer from two critical limitations in computational social science:

Loss of Heterogeneity: Aggregating data into a single corpus-level score obscures individual author-level variations. It cannot distinguish between a community where everyone holds a mild bias versus one where a small minority holds extreme bias.
Ecological Fallacy: Aggregate scores may reverse or cancel out effects that are consistently present at the individual level. For example, a corpus might show a neutral average association between "immigrants" and "crime" because it mixes users who use humanizing language with those who use slurs, masking the distinct trajectories of both groups.

There is a lack of methodology that treats individual user vectors as first-class objects for calibrated association testing, allowing researchers to quantify stance, bias, or stylistic alignment on a per-author basis without relying on supervised classifiers.

2. Methodology: POLAR

POLAR (Per-user On-axis Lexical Association Report) is a framework that performs association testing directly within the embedding space of a lightly adapted Masked Language Model (MLM). The pipeline consists of two phases:

Phase 1: User Vector Learning (Training)

User Tokenization: Each author $u$ is mapped to a unique, deterministic private token $t_u$ (e.g., usr + SHA1 hash). This token is added to the tokenizer vocabulary.
Data Preparation: Every post $x_i$ by user $u$ is prepended with their token ( $\tilde{x}_i = t_u \parallel x_i$ ).
Model Adaptation: Starting from a pre-trained model (e.g., BERT-base-uncased), the input embedding matrix is resized to include user tokens.
Training Objective: The model is fine-tuned using a standard Masked Language Modeling (MLM) loss with specific modifications:
- User-Aware Masking: The user token is masked with a higher probability than standard tokens to force the model to learn the user's linguistic context.
- Alignment Term: A small loss term nudges the user token's embedding vector toward the average representation of the non-user tokens in the post, ensuring the vector summarizes the user's typical style.
- Freeze-Unfreeze Schedule: A short training schedule stabilizes the learning of these new token vectors.
Output: Each user $u$ is represented by a normalized embedding vector $\hat{e}_u$ .

Phase 2: Association Testing (Inference)

Attribute Sets: Researchers define semantic axes as pairs of attribute sets $(A, B)$ (e.g., "Slurs" vs. "Neutral Terms"). These are mapped to embeddings via subword averaging.
Statistic Calculation: For a user $u$ $u$ and attribute sets $A$ $A$ and $B$ $B$ , POLAR computes cosine similarities between the user vector and the attribute vectors.
- The standardized effect size $s(u; A, B)$ is calculated as:
  $s(u; A, B) = \frac{\text{mean}(d_A) - \text{mean}(d_B)}{\text{sd}(d_{all})}$
  where $d_A$ and $d_B$ are the similarity scores to sets $A$ and $B$ , and $d_{all}$ is the pooled set of all similarities. This standardization ensures scale stability.
Significance Testing:
- Permutation Test: A Monte Carlo permutation test (default $M=2000$ ) shuffles the labels of the attribute sets to generate a null distribution.
- P-value: A two-sided p-value is estimated with add-one smoothing.
- Multiple Testing Control: The Benjamini–Hochberg (BH) procedure controls the False Discovery Rate (FDR) across all tested users and axes.

3. Key Contributions

Per-User Association Testing: POLAR is the first method to apply calibrated association testing directly to individual user embeddings, moving beyond aggregate corpus-level analysis.
Label-Free Diagnostics: The method requires no class labels (e.g., "bot" or "human") during the association testing phase. It produces interpretable effect sizes and p-values based solely on the geometry of the embedding space.
Modularity: The framework is modular; new attribute sets can be added or revised without retraining the underlying language model.
Interpretability: The sign of the score indicates the direction of alignment (e.g., toward "slurs" vs. "neutral"), and the magnitude indicates the strength of that alignment.

4. Experimental Results

The authors evaluated POLAR on two distinct datasets:

A. Bot-Human Twitter Benchmark (fox8-23)

Dataset: 1,140 AI-powered bot accounts and 1,140 human accounts.
Task: Distinguish bots from humans using per-user association scores on axes like "Scam/Finance vs. Daily Life" and "Newswire vs. Personal Experience."
Findings:
- POLAR cleanly separated bots from humans without using labels during training.
- Performance: A single-axis logistic regression achieved an AUROC of 0.961 (Scam/Finance axis). Combining axes did not significantly improve performance, suggesting individual axes capture strong stylistic signals.
- Visualization: PCA projections showed distinct clustering of bots and humans, with bots aligning strongly with "bot-speak" attributes (disclaimers, instructional scaffolding) and humans with colloquial markers.

B. Stormfront White-Supremacist Forum

Dataset: 10,568 sentences from a white-supremacist forum (2002–2017) with per-sentence user IDs.
Task: Quantify alignment with hateful lexicons (racial slurs, gender slurs, incel jargon) vs. neutral counterparts and track radicalization over time.
Findings:
- Strong Alignment: Users showed large positive effect sizes ( $s \approx 1.3$ ) for racial and gender slurs, with near-universal statistical significance after BH correction.
- Radicalization Drift: By tracking cumulative association scores over a user's posting history, POLAR revealed a rightward drift (increasing alignment with slur lexicons) for individual users over time. This serves as an interpretable, label-free indicator of progressive radicalization.
- Specificity: The signal was driven by domain-specific lexicons (slurs) rather than generic toxicity, as general "Toxicity vs. Civility" axes showed negligible shifts.

5. Significance and Implications

Computational Social Science: POLAR bridges the gap between NLP embedding techniques and social science inquiry. It allows researchers to study individual differences and within-group heterogeneity rather than relying on potentially misleading aggregate statistics.
Early Warning Systems: The ability to detect "drift" in user alignment with harmful lexicons offers a potential mechanism for identifying radicalization or bot behavior in real-time without requiring supervised training data.
Methodological Rigor: By providing permutation p-values and FDR control, POLAR moves beyond heuristic scoring to statistically grounded inference, making the results robust and reproducible.
Ethical Application: The authors emphasize the use of pseudonymous tokens and public datasets, ensuring the method is designed for research and diagnostics rather than surveillance or profiling of specific individuals.

In summary, POLAR provides a robust, statistically sound, and interpretable tool for analyzing how individual authors align with specific semantic frames, offering a new paradigm for detecting bias, bot activity, and radicalization in digital discourse.