🤖 AI

InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluation

To address the statistical instability and opacity of centralized LLM benchmarking, this paper proposes **InfiCoEvalChain**, a blockchain-based decentralized framework that leverages heterogeneous compute nodes and a reward-driven consensus mechanism to provide more stable, reliable, and transparent model evaluations.

Original authors: Yifan Yang, Jinjia Li, Kunxi Li, Puhao Zheng, Yuanyi Wang, Zheyan Qu, Yang Yu, Jianmin Wu, Ming Li, Hongxia Yang

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yifan Yang, Jinjia Li, Kunxi Li, Puhao Zheng, Yuanyi Wang, Zheyan Qu, Yang Yu, Jianmin Wu, Ming Li, Hongxia Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Fickle Exam" Problem

Imagine you are a student preparing for a massive final exam. You take a practice test on Monday and get a 95%. You take the exact same test on Tuesday, but because the room was slightly warmer, the lighting was different, and you were using a different brand of pen, you get an 85%.

In the world of Artificial Intelligence (AI), this is actually happening. Currently, when scientists want to see how "smart" an AI model is, they use "benchmarks" (standardized tests). But because AI generates answers using a bit of randomness (like rolling dice), and because different computers have different "temperatures" or hardware, the scores jump around wildly.

Right now, the gap between the "smartest" AI and the "second smartest" AI is often smaller than the error margin of the test itself. It’s like trying to rank the fastest runners in the world, but the stopwatch is broken and changes its reading every time you press start. We don't actually know who is winning; we just know who got a "lucky" reading.

The Solution: InfiCoEvalChain (The "Global Jury")

The researchers proposed a new system called InfiCoEvalChain. Instead of one big company (like Google or OpenAI) running the test in their own private lab (a "Black Box"), they want to turn the test into a Global, Decentralized Jury.

Here is how it works using three metaphors:

1. The Diverse Jury (Hardware & Parameter Diversity)

Instead of one professor grading a paper, imagine a jury of 1,000 people from all over the world. Some are sitting in snowy mountains, some are in tropical jungles, some are using high-end supercomputers, and some are using old laptops.
By having everyone grade the same AI at the same time, the "noise" (the weirdness caused by the environment) cancels itself out. If the AI is truly smart, it will perform well for everyone. If it just got "lucky" because of a specific computer setting, that luck won't hold up across the whole jury.

2. The Sealed Envelope (The Blockchain & Commit-Reveal)

To make sure no one cheats, they use Blockchain technology. Think of this like a "Sealed Envelope" system.

Step 1 (Commit): A juror grades the AI and puts their score in a sealed, locked envelope. They tell the world, "I have a score, and here is a digital fingerprint to prove it's locked."
Step 2 (Reveal): Only after everyone has submitted their envelopes does the jury open them all at once.
This prevents "copycatting," where one juror waits to see what the smartest person said and then just copies them to get a reward.

3. The "Fair Play" Reward (The Incentive Mechanism)

How do you keep people from lying or being lazy? You use a Smart Reward System.
Imagine a game show where you get paid based on how close your answer is to the "consensus" (the middle ground of the group). If you provide a score that is wildly different from everyone else (an outlier), the system assumes you are either a cheater or your computer is broken, and you get almost nothing. If you are part of the honest majority, you get a fair share of the prize.

The Result: From "Blurry" to "High-Definition"

The researchers tested this, and the results were massive.

Before, the scores were "blurry"—they fluctuated so much you couldn't trust them. After using this decentralized "Jury" method, the fluctuations dropped significantly (in one test, the error margin dropped from 1.67 down to 0.28).

In short: They have moved AI evaluation from a "shaky, private measurement" to a "rock-solid, public consensus." It’s like moving from a blurry, handheld camera photo to a crystal-clear, professional photograph. Now, when a leaderboard says "Model A is better than Model B," we can actually believe it.

Technical Summary: InfiCoEvalChain

InfiCoEvalChain is a proposed blockchain-based decentralized framework designed to revolutionize the evaluation of Large Language Models (LLMs). It addresses the fundamental instability and lack of transparency inherent in current centralized benchmarking practices.

1. The Problem: Statistical Fragility and Centralization

The authors identify three critical flaws in the current LLM evaluation landscape:

Statistical Fragility: Generative models are inherently stochastic. The paper reveals that the standard deviation of a single model's performance across repeated runs (e.g., 1.67 on HumanEval) can actually exceed the performance gap between the top-10 models on official leaderboards (0.91). This makes current rankings statistically unreliable.
Environmental Variance: Centralized evaluations often rely on specific hardware and inference configurations, introducing "noise" that does not reflect true model capability.
"Gray Areas" and Centralization: A few large entities control the evaluation process, leading to risks of data leakage, benchmark overfitting, and a lack of transparency (the "black box" problem).

2. Methodology: The CoEvalChain Framework

The authors propose a two-layer architecture to transform evaluation from a "centralized black box" into a "decentralized endorsement."

A. Collaborative Layer (Upper Layer):
This layer aggregates heterogeneous computing resources (GPUs like H800, A800, and consumer-grade RTX 5090s) from a global network of researchers and institutions. By running evaluations across diverse hardware and varying inference parameters (Temperature, Top-P, etc.), the framework statistically neutralizes stochastic noise.

B. Blockchain Foundational Layer (Lower Layer):
To ensure integrity in a distributed network, the framework utilizes blockchain technology:

Sybil Resistance: A token-based staking mechanism requires nodes to have "skin in the game" before participating.
Consensus via Game Theory: The system employs a Schelling Point (Focal Point) mechanism. Rational nodes are incentivized to converge on the "truth" (the most salient score) to maximize rewards.
Commit-Reveal Scheme: To prevent "free-riding" (nodes simply copying others), a cryptographic two-phase protocol is used. Evaluators first submit a hash of their result (Commit) and only reveal the actual score (Reveal) after the window closes.
Robust Incentivization: Instead of using a simple mean, which is susceptible to outliers, the framework uses a proximity-based reward system centered on the median. Rewards are distributed using a Gaussian decay function, ensuring that nodes providing results far from the consensus receive minimal or no rewards.

3. Key Contributions

First Formal Definition: The paper provides the first formal definition of a blockchain-based decentralized mechanism specifically for generative model evaluation.
Statistical Framework: It establishes a probabilistic foundation (using the Central Limit Theorem) to determine the required number of independent observations ( $n$ ) needed to achieve high-confidence results.
Functional Prototype: The authors have implemented a fully operational platform and intend to release it to the community.

4. Experimental Results

The framework was tested across various model scales (from 7B to ultra-large models like DeepSeek-V3.2 and Gemini-2.5-Flash) and diverse benchmarks (GSM8K, HumanEval, GPQA-Diamond, etc.).

Significant Stability Gains: The decentralized framework reduced the standard deviation of model runs on HumanEval from 1.67 to 0.28.
Precision in Difficult Tasks: In high-difficulty benchmarks like GPQA-Diamond, where volatility is highest, the framework drastically tightened confidence intervals (e.g., reducing standard deviation for Qwen-3-14B from 4.25 to 0.92).
Effective Incentivization: Simulation results showed a Gaussian-like reward distribution, successfully identifying and penalizing outliers (approximately 3% of nodes) while maintaining community stability.
Data Partitioning: The proposed stratified sampling method successfully maintained uniform task difficulty across different data subsets.

5. Significance and Impact

InfiCoEvalChain shifts the paradigm of AI assessment. By moving toward a decentralized endorsement ecosystem, it provides:

Reliable Metrics: High-confidence leaderboards that allow developers to distinguish genuine architectural breakthroughs from environmental noise.
Democratization: It breaks the monopoly of large corporations over AI evaluation, inviting global participation.
Transparency: The immutable blockchain ledger ensures that every evaluation is auditable, preventing data manipulation and fostering trust in the AI research community.