💬 NLP

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

The paper introduces PoQ-Judge, a multi-architecture framework that trains reference-free judge models to evaluate decentralized LLM inference quality, achieving strong correlation with ground-truth proxies and significant cost reductions through online calibration and cascade evaluation.

Original authors: Arther Tian, Alex Ding, Frank Chen, Simon Wu, Aaron Chan

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arther Tian, Alex Ding, Frank Chen, Simon Wu, Aaron Chan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, global team of volunteers trying to answer questions or write summaries using Artificial Intelligence. Because everyone is working from different computers with different speeds, the system needs a way to decide: "Is this answer actually good?" and "Who deserves a reward for doing the work?"

In the past, the system had a major problem: to know if an answer was good, it needed to compare it to a "perfect" answer (like a teacher's answer key). But in a live, real-time chat, no one has the answer key. The system was stuck.

This paper introduces PoQ-Judge, a new solution that acts like a super-smart, lightning-fast referee who can grade an answer without ever seeing the answer key.

Here is how it works, broken down into simple parts:

1. The Problem: The Missing Answer Key

Imagine a teacher grading a student's essay. Usually, the teacher checks the essay against a "model answer" to see if it's right.

The Old Way: The system tried to use "model answers" to grade everything. But in a live chat, the model answer doesn't exist yet.
The Result: The system was blind. It couldn't tell if a response was brilliant or nonsense without that reference.

2. The Solution: The "PoQ-Judge" Referees

The authors built three special AI "referees" (called Judge Models) that are trained to look at a Question and an Answer and give it a score from 0 to 10, just by reading them. They don't need a reference answer.

Think of these referees as three different types of workers, each with a different speed and skill level:

The Speedster (TextCNN):
- What it is: A tiny, super-fast worker.
- Speed: It grades an answer in less than a blink of an eye (sub-millisecond).
- Skill: It's good at spotting obvious nonsense, but it's not a deep thinker. It's like a security guard who quickly checks if a door is locked.
- Use: Perfect for checking thousands of answers quickly when you are on a tight budget.
The Balanced Worker (MiniLM):
- What it is: A mid-sized worker.
- Speed: Takes a tiny fraction of a second (about 13 milliseconds).
- Skill: It's a good all-rounder. It understands the meaning well enough for most situations.
The Expert (DeBERTa):
- What it is: A large, highly trained expert.
- Speed: Takes a bit longer (about 15 milliseconds), but still very fast.
- Skill: This is the best one. It understands nuance and context very deeply. The paper found this expert was actually better at grading than the old systems that did have answer keys.

3. How They Learned Their Job

You can't just give a referee a rulebook and expect them to be perfect. The authors trained these referees in two steps:

School (Pre-training): They studied a massive library of 45,000 examples where a super-smart AI (GPT-4) had already graded answers. This taught the referees what "good" generally looks like.
Internship (Fine-tuning): They practiced on specific tasks (like answering questions and summarizing news) using 1,400 examples labeled by the same super-smart AI. This made them experts in the specific type of work the network does.

4. The "Cascade" Strategy: Saving Money

The system is smart about money. It doesn't always hire the expensive Expert (DeBERTa) for every single job.

The Cascade Protocol: Imagine a funnel.
- First, the Speedster checks the answer. If the answer is obviously terrible (or obviously perfect), the system stops there and saves money.
- If the Speedster is unsure, the answer gets passed to the Balanced Worker or the Expert.
The Result: This strategy saved the system up to 72% in costs because most answers didn't need the expensive expert to be graded.

5. The Big Catch: It Depends on the Task

The paper found a surprising twist. The referees were amazing at grading answers to questions (like "Who was the first president?"). They got a score of 0.83 out of 1.0 accuracy.

However, they struggled with summarizing long texts (like "Summarize this news article"). Their accuracy dropped to 0.20.
Why? The "ground truth" (the way they measured success) for summaries is flawed. It's like trying to grade a painting based only on how many red pixels are in it. The referees learned to play by the flawed rules, so they couldn't do well on summaries.

Summary of the Win

The paper proves that you don't need an answer key to grade AI answers effectively. By training small, specialized "referees," the system can:

Grade answers in real-time without waiting for a reference.
Be just as accurate (or better) than old methods that did have reference keys.
Save a huge amount of money by using fast, cheap referees for easy jobs and saving the experts for the hard ones.

The only thing holding it back right now is that the "grading rules" for summarizing tasks need to be improved so the referees can learn to do that job better too.

Technical Summary: PoQ-Judge

Problem Statement
Decentralized Large Language Model (LLM) inference networks rely on Proof of Quality (PoQ) to verify outputs and allocate rewards without costly cryptographic proofs. While prior work established cost-aware incentives, adversarial robustness, and multi-dimensional quality scoring, a critical deployment gap remained: the most reliable quality dimension—semantic similarity—requires a ground-truth reference answer. In live decentralized networks, such references are unavailable. Existing reference-free evaluators (e.g., pre-trained NLI cross-encoders) correlated poorly or negatively with ground-truth quality, while "LLM-as-a-Judge" approaches using billion-parameter models were too latency-heavy and expensive for the high-throughput, cost-constrained requirements of PoQ consensus rounds.

Methodology
The paper introduces PoQ-Judge, a multi-architecture framework designed to train lightweight, reference-free judge models specifically for decentralized inference. The methodology comprises three core components:

Multi-Architecture Design: The authors propose three judge models spanning a quality–cost Pareto frontier:
- TextCNN Judge (~10M params): A convolutional architecture offering sub-millisecond latency (~1ms), suitable for high-throughput or cost-constrained tiers.
- MiniLM Judge (22M params): A cross-encoder based on a distilled Transformer, balancing quality and latency (~13ms).
- DeBERTa Judge (184M params): A disentangled-attention encoder targeting maximum accuracy (~15ms).
  All models output a scalar quality score $s_\theta(q, y) \in [0, 10]$ for a query–output pair without accessing a reference answer.
Two-Stage Training Pipeline: To overcome data scarcity in the specific PoQ domain, the models undergo:
- Stage 1 (Pre-training): Training on the UltraFeedback corpus (45k samples) to learn general evaluation patterns and instruction-following quality.
- Stage 2 (Fine-tuning): Adaptation to the target PoQ task distribution using GPT-4o-mini labeled data (1,400 training samples) covering Question Answering (QA) and Summarization. The objective is Mean Squared Error (MSE) regression against the labeled scores.
Integration and Deployment Mechanisms:
- Composite Scoring: The trained judges are integrated as a new "reference-free" dimension within a multi-dimensional scoring framework, alongside structural priors and optional reference-based dimensions.
- Online Calibration: The framework supports online dimension weight adjustment via Exponential Moving Average (EMA), Multi-Armed Bandit (UCB), and Gradient Descent strategies to adapt to shifting task distributions.
- Cascade Evaluation: A cost-aware protocol routes outputs through evaluation layers (Structural $\to$ Lightweight Judge $\to$ Full Evaluation), stopping early when confidence thresholds are met to save costs.

Key Results
Experiments were conducted on a held-out test set ( $n=300$ ) spanning QA and summarization tasks, using token-level F1 against ground-truth as the proxy.

Judge Performance: The DeBERTa Judge achieved a Pearson correlation of 0.747 with the ground-truth proxy, exceeding the best reference-based baseline from prior work (STS-stsb: 0.647). The 95% bootstrap confidence interval [0.663, 0.816] did not overlap with the upper bound of the best reference-based evaluator.
Reference-Free Composite Scoring: When integrated into a composite score without reference answers, the system achieved a Pearson correlation of 0.645, statistically matching the best single reference-based evaluator (STS-stsb: 0.647) without requiring reference data.
Task Dependence: Performance varied sharply by task. On QA, the DeBERTa judge reached 0.830 Pearson; on Summarization, it dropped to 0.199. The authors attribute this primarily to the limitations of the token-level F1 ground-truth proxy for summarization, rather than model capacity.
Cost Savings: The cascade evaluation protocol achieved up to 72.7% cost savings in low-budget regimes by relying on structural priors, with only modest quality reduction.
Online Calibration: Gradient-based calibration successfully identified semantic quality as the dominant dimension (learning a weight 4.7 $\times$ the initial value) and suppressed dimensions with negative correlations (e.g., query–output alignment), recovering insights from offline ablation studies.

Significance and Claims
The paper claims that PoQ-Judge successfully bridges the deployment gap between offline multi-dimensional scoring and live decentralized inference. By demonstrating that compact encoder models trained via a two-stage pipeline can outperform reference-based evaluators without access to ground-truth answers, the framework eliminates the need for maintaining and distributing reference answer databases in decentralized networks.

The authors position this work as a complete pipeline extension to their prior PoQ research (cost-aware rewards, robust aggregation, and multi-dimensional scoring). They emphasize that while the framework validates the viability of reference-free evaluation for PoQ incentives, the primary open challenge remains the quality of the ground-truth proxy, particularly for non-extractive tasks like summarization. The paper does not claim to solve general LLM evaluation but specifically targets the efficiency and feasibility constraints of decentralized inference networks.