From Programs to Predictions: A Scalable, Multilingual Platform for Automated Evaluation of Machine-Learning Olympiads

Imagine you're at a massive, high-stakes cooking competition, but instead of handing in a finished dish to be tasted, the contestants are sending you a secret recipe card with a list of predicted flavors. In the old days, computer contests were like a strict taste-test: you submitted a program, the judge ran it, and if the output didn't match the answer key exactly, you failed. But in this new world of "Machine Learning Olympiads," the rules have changed. The judges don't care if the code runs perfectly; they care if the predictions are close enough to the truth, using weird, custom-made scoring rules that change for every single problem.

Enter MLCompete, a digital platform built to handle this chaotic, high-speed flavor-challenge. It's the official referee for Romania's National AI Olympiad, and it's doing something no other system quite does: it lets thousands of students compete, submit their "flavor predictions," and get scored instantly, even when the scoring rules are as unique as snowflakes.

The Big Shift: From Running Code to Grading Guesses

The paper argues that you can't just take an old-school computer judge and tweak it for this. It's like trying to use a stopwatch to measure the taste of soup. The old systems wait for a program to finish running and spit out an answer. MLCompete, however, realizes that in AI contests, the heavy lifting happens before the submission. Students train their models elsewhere (or on the platform), then upload a file full of predictions. The platform's job is to take that file, compare it to a hidden "answer key" (ground truth), and calculate a score using a metric that might be specific to that one problem.

The authors explicitly rule out the idea that this is just a "bigger version" of a standard programming contest. They say the infrastructure needs to be fundamentally different because the deliverable isn't a program; it's a set of predictions, and the scoring isn't "right or wrong," it's a sliding scale of how good the guess is.

How It Works: The Assembly Line

Think of MLCompete as a super-efficient, automated factory line.

1. The Drop-Off: When a student uploads their prediction file, the system doesn't make them wait. It's like a smart mailroom that instantly stamps your package "Received" and says, "We'll get to this in a jiffy." This is the asynchronous part. The student can go play a game while the real work happens in the background.
2. The Queue: The submission gets dropped into a digital waiting line (a message queue). This is the secret sauce. It means if 1,000 students submit at the exact same second, the system doesn't crash; it just lines them up.
3. The Scoring Crew: A team of invisible workers (called "evaluators") grabs the next file from the line, runs the specific scoring script (which could be a standard math formula or a custom script written by the problem author), and calculates the score.
4. The Result: Once the score is ready, the system pushes it back to the student's screen in near real-time.

The paper shows that this setup is incredibly fast. In a 15-day window, the system handled over 8.1 million requests. The backend (the brain of the operation) was so fast that 95% of the time, it answered in less than 255 milliseconds. That's faster than a blink of an eye.

The "Two-Phase" Scoreboard

Here's a tricky part the paper explains with a clever trick to stop cheating. Imagine a leaderboard that shows you how you're doing right now, but it's only based on a small, public slice of the test questions. This keeps students engaged. But the real final ranking is based on a secret, private slice of questions that no one has seen.

The platform calculates both scores in one go. It prevents students from "gaming the system" by submitting thousands of times to see which one accidentally hits the secret answers. To win, a student must explicitly pick their two best submissions before the deadline. The system then takes the better of those two on the secret leaderboard. This forces students to trust their best work rather than spamming the system.

The "Safe Room" for Code

Sometimes, students need to run their own code directly on the platform's super-computers to generate their predictions. This is dangerous because you don't want a student's buggy code to steal data or crash the whole system.

The paper describes a "defense-in-depth" security model. For the most advanced stage (the National Team Selection Camp), students get access to powerful NVIDIA H200 GPUs. But here's the magic: the platform uses a technology called MIG (Multi-Instance GPU) to slice one giant GPU into five smaller, isolated pieces. Each student gets their own tiny slice of memory and computing power. It's like giving every contestant their own private, soundproof booth in a giant library. Even if one student's code goes haywire, it can't touch anyone else's data or the rest of the computer.

To make sure they aren't cheating by looking up answers online, the platform uses a "whitelisting proxy." It's like a strict librarian who only lets you read books from a specific, pre-approved list. During the contest, students can only talk to the platform, their identity provider, and the Jupyter coding environment. The internet, public model hubs, and random websites are completely blocked.

The Numbers: How Well Did It Work?

The authors didn't just build this; they ran it through the fire of a real national competition.

• Reliability: The system was up and running almost perfectly. The "server error" rate was a tiny 0.007%. That means out of every 10,000 requests, only about 7 failed.
• Cost: They managed all this on a single, modest computer server (a "commodity K3s node") for the steady, everyday traffic. It's like running a massive theme park with a single ticket booth that only opens up more windows when the crowd gets huge.
• The Spike: During the National Team Selection Camp, the traffic jumped 3 to 6 times higher than normal. The system absorbed this spike without slowing down or crashing, thanks to its ability to automatically add more "workers" when the line got long.
• Scalability: The authors ran simulations (computer models, not live tests) to see what would happen if hundreds of students submitted at once. They found that with a pool of just 10 workers, the system could handle up to 5 submissions per second with almost zero wait time. If the line got longer, the system would just add more workers automatically.

What It's Not

The paper is careful to say what it hasn't proven yet. They admit that their live data only covers a 15-day window, which included a "modest" contest spike but missed the massive "spring" stages where hundreds of students compete. So, while the system handled the team camp spike perfectly, the authors suggest (rather than prove) that it would scale even further for the biggest events. They also note that while they have strong security, they are still working on even tougher "kernel-level" sandboxing to make the code execution absolutely bulletproof against the most determined hackers.

The Takeaway

MLCompete proves that you can build a massive, multilingual, secure, and cheap platform for AI competitions by treating predictions like mail and scoring like a factory assembly line. It's not a magic wand that solves all AI problems, but it's a solid, working blueprint that lets students focus on learning and competing, rather than worrying if the computer will crash. As the paper concludes, the architecture works, and the data backs it up, offering a new way to run the future of AI education.