💬 NLP

Pearmut: Human Evaluation of Translation Made Trivial

The paper introduces Pearmut, a lightweight and feature-rich platform designed to simplify the setup and execution of human evaluation for multilingual NLP tasks, particularly machine translation, by removing engineering barriers and supporting standard protocols to make reliable human assessment a routine part of model development.

Original authors: Vilém Zouhar, Tom Kocmi

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vilém Zouhar, Tom Kocmi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to invent the perfect new recipe. You have a robot assistant that can taste your dish and give you a score based on a complex formula. The robot is fast, cheap, and always ready. But sometimes, the robot gets it wrong. It might say a dish is "perfect" because it has the right ingredients, even though it tastes like cardboard.

In the world of Artificial Intelligence (specifically translation), researchers rely heavily on these "robot taste testers" (automatic metrics) to judge their work. But the gold standard for knowing if a translation is actually good is a human tasting it.

The problem? Getting a human to taste-test is a nightmare. It's slow, expensive, and the tools to organize it are like trying to build a kitchen using a hammer and a screwdriver. Most researchers skip the human taste test because it's too hard to set up.

Enter Pearmut.

What is Pearmut?

Think of Pearmut as a smart, all-in-one food delivery app for translation testing.

Before Pearmut, if you wanted to organize a human taste test, you had to:

Build your own kitchen (write complex code).
Hire the chefs (find people who speak the languages).
Create your own menus and scorecards (design the evaluation forms).
Manually collect the scores and do the math.

Pearmut does all of that for you. It's a lightweight, easy-to-install tool that turns the complex, scary process of human evaluation into something as simple as clicking "Play" on a video.

How Does It Work? (The Metaphors)

1. The "Magic Link" Invitation
Usually, getting people to do a task is like herding cats. You have to chase them, remind them, and hope they show up.
Pearmut uses "Magic Links." You generate a special link, send it to your team (or a crowd of workers), and poof! They are instantly logged in and ready to work. No passwords, no confusing sign-up forms. It's like sending a VIP pass that opens the door automatically.

2. The "Side-by-Side" Tasting
When humans judge translations, they often need to see the original text and multiple versions of the translation at the same time to spot the differences.
Old tools made you switch back and forth between screens, like flipping through a book with your eyes closed. Pearmut shows you the Original, Version A, and Version B all on one screen, side-by-side. It's like having three plates of food in front of you at once so you can compare them instantly.

3. The "Smart Manager" (Dynamic Assignment)
Imagine you have 100 dishes to taste, but only 10 tasters.

The Old Way: You give everyone the same 10 dishes, then the next 10, randomly. You might waste time tasting a terrible dish when you already know it's bad.
The Pearmut Way: It's a smart manager. If the system notices that "Dish A" is consistently rated as the best, it stops showing the terrible dishes and focuses the tasters on comparing the top contenders. It saves time and money by only asking for opinions on the things that actually matter.

4. The "Safety Net" (Attention Checks)
Sometimes, people get bored and just click random buttons to finish quickly.
Pearmut has built-in traps (called attention checks). It might include a test question like, "If you are reading this, please click the red button." If a worker misses it, the system knows they aren't paying attention and flags their work. It's like a teacher giving a pop quiz to make sure students are actually reading the instructions.

Why Should We Care?

The authors of the paper tested Pearmut against other tools and found:

It's Faster: Setting up a test took researchers about 11 minutes with Pearmut, compared to over 20 minutes (or failing completely) with other tools.
It's Easier: Even an AI coding agent (a robot programmer) could set up Pearmut automatically. It couldn't do that with the older, clunky tools.
It's Better: The humans using Pearmut felt less stressed, worked faster, and gave more accurate feedback because the interface was designed specifically for them, not for a generic database.

The Bottom Line

For a long time, human evaluation in AI was like trying to build a house with a Swiss Army knife. It could be done, but it was painful and slow.

Pearmut is the power drill. It doesn't change the goal (building a great house/translation), but it removes the friction, the engineering headaches, and the fear. It makes human evaluation so easy that researchers can finally stop guessing and start knowing exactly how good their AI really is.

In short: Pearmut makes "asking a human" as easy as "asking a computer." And that's a game-changer.

1. Problem Statement

Human evaluation is the "gold standard" for assessing Multilingual Natural Language Processing (NLP), particularly Machine Translation (MT). However, it is frequently skipped in research and industry due to:

High Engineering Overhead: Existing tools (e.g., Appraise, Label Studio) are often difficult to deploy, require complex configuration, or lack specific features for translation tasks.
Operational Complexity: Setting up campaigns, managing annotator assignments, and ensuring data quality (via attention checks) is time-consuming.
Scarcity of Reporting: A review of 82 ACL 2025 translation papers found that 68% did not report any human evaluation. When used, protocols are often ad-hoc, non-standard, and irreproducible.
Tool Limitations: General-purpose tools lack domain-specific logic (e.g., contrastive evaluation, span-level error marking), while specialized tools (like Appraise) are hard to modify or deploy.

2. Methodology: The Pearmut Platform

The authors introduce Pearmut, a lightweight, pip-installable Python package designed to make human evaluation as easy to set up as automatic metrics (e.g., SacreBLEU).

Core Architecture

Tech Stack: Built with FastAPI (Python) for the backend and TypeScript/jQuery for the frontend.
Data Storage: Uses an in-memory store for low-latency access, backed by an append-only log file on disk for durability. It avoids heavy database management systems to ensure speed and simplicity.
Deployment: Requires only three commands: pip install, add campaign, and run.

Key Features & Protocols

Pearmut supports standard evaluation protocols and extends them with modern features:

Supported Protocols:
- Direct Assessment (DA): 0–100 scale.
- Error Span Annotation (ESA): Marking error spans with severity (minor/major) and a final score.
- Multidimensional Quality Metrics (MQM): Granular error categorization.
- ESAAI (Pre-filled): AI pre-annotates error spans for human post-editing, reducing annotator effort.
Document-Level Context: Supports contrastive evaluation where multiple model outputs are shown side-by-side with the source, extending beyond simple pairwise comparison.
Assignment Strategies:
- Task-based: Pre-assigns items to users (rigid).
- Single-stream: Randomly assigns unannotated items (high throughput, less control).
- Dynamic Assignment: Uses an $\epsilon$ -greedy strategy (Multi-Armed Bandit) to prioritize evaluating top-performing models, optimizing budget for finding the best system.
Quality Control:
- Attention Checks & Tutorials: Interactive tasks to ensure annotator focus; can block progress until conditions are met.
- Redundancy: Allows re-doing annotations without destroying collected data.
Multimodal Support: Handles text, audio, video, and custom HTML elements.
Dashboard: Provides real-time monitoring of progress, attention check success rates, and statistical analysis (e.g., two-sided t-tests for model ranking).

3. Key Contributions

Accessibility: A "zero-friction" setup process that lowers the barrier to entry for human evaluation, aiming to make it a routine part of model development.
Domain Specialization: Unlike general tools, Pearmut is engineered specifically for translation, supporting best practices like ESA, MQM, and document-level contrastive evaluation out of the box.
Statistical Rigor: Built-in statistical testing (t-tests) and dashboard visualization to promote standardized scientific reporting.
AI-Assisted Workflow Compatibility: Designed to be easily scripted and integrated into modern AI-assisted research pipelines (verified via LLM coding agent tests).

4. Empirical Results

The authors conducted two case studies to validate Pearmut:

Case Study 1: Researchers (Setup & Usability)

Participants: 5 NLP practitioners.
Task: Set up evaluation campaigns using Pearmut, Appraise, Potato, Label Studio, and Factgenie.
Findings:
- Speed: Pearmut was the fastest to set up (11 minutes on average) compared to Appraise (25m) and Label Studio (19m).
- Success Rate: Researchers successfully completed the setup with Pearmut 100% of the time. Other tools saw multiple failures or researchers giving up.
- LLM Agent Test: An LLM coding agent successfully wrote setup scripts for Pearmut and Label Studio but failed on Appraise and Factgenie due to non-standard architectures.

Case Study 2: Annotators (Performance & Satisfaction)

Participants: 10 bilingual speakers.
Task: Annotate 540 translation outputs (English $\to$ 8 languages) using ESA protocol in both Pearmut and Appraise.
Findings:
- Efficiency: Annotators were ~15% faster with Pearmut (124s/item vs. 145s/item in Appraise).
- Quality: Inter-annotator agreement and model separation scores were comparable, indicating Pearmut does not sacrifice data quality for speed.
- Satisfaction: Annotators rated Pearmut significantly higher in speed (8.3/10 vs 6.3), clarity (8.3/10 vs 6.0), and effort (7.8/10 vs 6.0).
- Server Performance: Pearmut responded 3x faster to requests than Appraise and could theoretically support 2,000+ concurrent users on cheap hardware, far exceeding the ~158 concurrent users in the WMT 2025 campaign.

5. Significance

Paradigm Shift: Pearmut aims to shift human evaluation from an "occasional, high-effort effort" to a "practical, routine component" of NLP research.
Reproducibility: By enforcing standard protocols (ESA, MQM) and providing easy export of JSON data, it addresses the reproducibility crisis in translation evaluation.
Adoption: The tool is already being adopted for diagnosing speech model errors and is selected for the WMT 2026 General Translation Shared Task, signaling its acceptance by the community.
Ethical & Practical Balance: While it requires basic engineering skills (CLI, JSON config), it significantly reduces the engineering burden compared to alternatives, making high-quality human evaluation accessible to a broader range of researchers and practitioners.

In conclusion, Pearmut successfully bridges the gap between the theoretical necessity of human evaluation and the practical constraints of time and engineering resources, providing a robust, fast, and user-friendly platform for the multilingual NLP community.