React-ing to Grace Hopper 200: Five Open-Weights Coding Models, One React Native App, One GH200, One Weekend

Imagine you are a chef trying to build the perfect kangaroo-counting app for a weekend cooking competition. You have five different, highly-rated "recipe books" (AI models) to choose from. These books are famous because they scored incredibly high on a standardized test called SWE-Bench (think of it as a Michelin-star rating for coding skills).

You also have a super-fast, massive kitchen oven (an NVIDIA GH200 computer) that can cook anything instantly.

The goal? To see which recipe book actually produces a working app that you can open, log into, count kangaroos, and run on a website, all without you having to fix anything.

Here is the twist: The books with the highest Michelin stars (SWE-Bench scores) didn't win the competition. In fact, the "underdog" book won.

Here is the story of what happened, broken down simply:

1. The Unexpected Winner: The "Small" Book

The winner was Kimi-K2.5, but specifically a version that had been "compressed" (quantized) to fit on the oven.

• The Analogy: Imagine a massive encyclopedia that was shrunk down to a pocket guide. Even though it's smaller, it understood your instructions perfectly.
• The Result: It built an app that worked immediately. You could log in, count kangaroos, save the history, and even run it on a web browser. It was the only one that followed the rules exactly.

2. The "Star" That Failed: The Over-Engineer

The runner-up in the official rankings was GLM-5.1. It had the highest test scores.

• The Analogy: This chef was so obsessed with being "professional" that when you asked for a simple sandwich, they built a full industrial factory to make it.
• The Result: They built a beautiful app, but it required you to set up a massive, complex cloud database (Firebase) just to log in. Since you didn't have that database set up, the app wouldn't even start. It was technically "correct" but practically useless for your weekend project.

3. The "Star" That Got Lost in Translation

Another top contender, DeepSeek-V3.2, had a funny glitch.

• The Analogy: This chef started talking to themselves ("Let's start with the bread...") and accidentally wrote those thoughts inside the label of the bread bag.
• The Result: The computer got confused. It tried to save the main file (App.js) into a folder named "Let's start with App.js: [thinking tag]". Because the file was in the wrong place, the whole app fell apart. The model was smart, but it couldn't separate its "thinking" from its "doing."

4. The Silent Killer: The "Web" Problem

Almost every chef made the same mistake. They used a tool to show pop-up warnings (like "Are you sure you want to delete?") that works great on a phone but does nothing on a website.

• The Analogy: It's like a chef using a smoke alarm to signal "Dinner is ready" in a restaurant that only has electric lights. The alarm goes off, but nobody hears it.
• The Result: The apps looked perfect, but when you tried to use them on a web browser, the buttons did nothing. The models hadn't been trained enough on the difference between "phone apps" and "website apps."

The Big Lesson: Benchmarks Lie

The paper's main point is that test scores don't tell the whole story.

• The Test (SWE-Bench): Measures how well a model can fix small, isolated code snippets. It's like testing a chef on how well they can chop an onion.
• The Real World: Requires building a whole meal, understanding the customer's specific needs, and making sure the kitchen tools actually work together.

The "underdog" model (Kimi) won because it was better at listening to the whole prompt and understanding the context, not just because it had the most "active brain power" (parameters).

The Hardware Twist: Size Isn't Everything

The paper also looked at the "kitchen" (the computer hardware).

• The "Scale School": These are giant models that need massive, expensive servers (like the GH200) to run. They are heavy and slow to load.
• The "Efficiency School": These are smaller, smarter models that run fast on standard computers (like a Mac Studio).
• The Finding: You can get the same "cooking quality" with a small, efficient model running on a $10,000 computer as you can with a giant model running on a $75,000 server. For most people, the smaller, faster model is the better deal.

Summary

If you are building a real product today:

1. Don't just trust the test scores. A model might ace the test but fail the real job.
2. Watch out for "Thinking" leaks. Some models talk too much while they work, which can confuse the tools that save their files.
3. Check the platform. Just because code works on a phone doesn't mean it works on the web.
4. Smaller can be better. You don't always need the biggest, most expensive computer to get the best results.

In short: The model that listened best won, not the one that was rated highest.