Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models

1. Problem Statement

Competitive programming, particularly at the level of the International Olympiad in Informatics (IOI), serves as a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of Large Language Models (LLMs). While proprietary models (e.g., OpenAI's o1-ioi, o3) have demonstrated gold-medal-level performance, these results often rely on undisclosed methods and closed systems.

Key Challenges:

• Open-Weight Gap: Open-weight models (e.g., DeepSeek-R1, Qwen) currently lag behind proprietary models on complex IOI problems.
• Submission Constraints: IOI limits contestants to 50 submissions per problem, making the selection strategy critical. Generating many solutions is useless if the system cannot efficiently identify and submit the correct ones within the budget.
• Lack of Reproducibility: Previous high-performance results from proprietary systems lack transparency regarding their test-time compute (TTC) pipelines.

2. Methodology: GENCLUSTER Framework

The authors propose GENCLUSTER, a scalable, reproducible test-time compute framework designed to maximize the performance of open-weight models under strict submission constraints. The pipeline consists of four distinct stages (illustrated in Figure 1 of the paper):

A. Parallel Candidate Generation

• The system generates a massive pool of candidate solutions ($K$) for each subtask in parallel.
• Filtering: Solutions that fail to parse or compile are immediately discarded.
• Scale: In experiments, up to 5,000 candidates were generated per subtask.

B. Behavioral Clustering

To manage the large pool of candidates, solutions are grouped based on behavioral similarity rather than code similarity.

• Test Generation: The LLM generates randomized test inputs and independent validators (C++ programs) to ensure inputs meet subtask constraints.
• Validation: Inputs are accepted only if $\ge 75\%$ of validators approve them.
• Clustering: All candidate solutions are executed on the validated test cases. Solutions producing identical outputs across all test cases are grouped into the same cluster.
• Pruning: Clusters containing solutions that produce empty outputs (due to runtime errors) are removed.

C. Ranking with Tournament

Instead of simple voting, the framework uses a round-robin tournament to rank clusters:

• Representative Selection: From each cluster, the solution with the longest reasoning trace is selected as the representative.
• Pairwise Comparison: Representatives from two different clusters are pitted against each other. An LLM (acting as a judge) compares the code and reasoning traces to select a winner.
• Ranking: Clusters are ranked based on the number of "wins" their representative achieves. To mitigate recency bias, the order of presentation in pairs is randomized.

D. Round-Robin Submission Strategy

The system must adhere to the 50-submission limit per problem:

• Order: Submissions start with the most difficult subtasks (final subtasks).
• Selection: The system iterates through ranked clusters, submitting one solution from each cluster in a round-robin fashion.
• Termination: Once a subtask is solved (max score achieved), the system skips remaining clusters for that subtask and moves to the next.

3. Key Contributions

1. First Open-Weight Gold Medal: The paper demonstrates, for the first time, that an open-weight model (gpt-oss-120b) combined with a transparent TTC strategy can achieve Gold Medal performance at IOI 2025.
2. Scalable Framework: GENCLUSTER introduces a novel pipeline combining behavioral clustering, tournament-based ranking, and budget-aware submission strategies, bridging the gap between open and closed systems.
3. Reproducibility: Unlike proprietary claims, the authors provide a fully specified, reproducible methodology and plan to release the code publicly.
4. Scaling Laws: The study establishes that performance scales consistently with compute (number of generations), proving that open models can surpass proprietary ones if given sufficient test-time compute resources.

4. Experimental Results

The authors evaluated top open-weight models (gpt-oss-120b, gpt-oss-20b, DeepSeek-R1-0528, Qwen3-235B-A22B) on the IOI 2025 benchmark.

• Model Performance:
  • gpt-oss-120b significantly outperformed all other models, showing the strongest scaling properties as the number of generations increased.
  • Other models (DeepSeek, Qwen) showed diminishing returns or saturation at higher generation counts.
• Impact of GENCLUSTER:
  • With 5,000 generations per subtask and the GENCLUSTER pipeline, gpt-oss-120b achieved a score of 446.75.
  • This score exceeds the Gold Medal threshold (438.3), marking the first open-weight gold medal.
  • Comparison: While OpenAI's proprietary models reportedly scored higher (e.g., ~533), the gap is narrowing, and the open approach is fully transparent.
• Ablation Studies:
  • Tournament vs. Baselines: GENCLUSTER (Tournament ranking) scored 446.75, significantly outperforming baselines like random selection (300.10) or simple majority voting (314.22).
  • Representative Selection: Choosing the solution with the longest reasoning trace as the cluster representative proved superior to random selection.
  • Test Case Size: Increasing test cases improved cluster purity (F1-score) but increased the number of clusters, creating a trade-off in selection difficulty.
  • Token Limits: gpt-oss-120b continued to improve up to 120K tokens, whereas other models saturated earlier.

5. Significance and Limitations

Significance:

• Democratization: Proves that high-level reasoning capabilities are not exclusive to proprietary, closed-source models but can be achieved through scalable compute and clever engineering.
• Benchmarking: Sets a new standard for evaluating LLMs on complex reasoning tasks using open-weight models.
• Methodological Shift: Highlights that "test-time compute" (generating and filtering many solutions) is a more effective lever for complex problems than simply increasing model training size.

Limitations:

• Compute Cost: The approach is resource-intensive, requiring ~7.3 billion tokens for generation and another ~7.3 billion for the ranking tournament per benchmark run.
• Synthetic Test Blind Spots: Reliance on LLM-generated test cases may miss rare corner cases, leading to incorrect clustering.
• LLM-as-Judge Noise: The tournament ranking relies on an LLM to judge code correctness, which can be influenced by superficial features (style, length) rather than actual logic.
• Reasoning Length Heuristic: While longer reasoning traces correlated with correctness in this study, they can also indicate confusion or verbosity, making the heuristic imperfect.

Conclusion

The paper establishes that GENCLUSTER enables open-weight models to reach the pinnacle of competitive programming (IOI Gold). By scaling test-time compute and employing a sophisticated selection pipeline, the authors have created a reproducible baseline that narrows the performance gap between open and proprietary AI systems, suggesting a future where reasoning capabilities are driven more by inference strategies than just model weights.