OpenCoderRank: Personalized Technical Assessments with Generative AI

1. Problem Statement

The paper addresses a critical dichotomy in the era of Large Language Models (LLMs) regarding technical assessments:

• For Problem Setters (Educators/Recruiters): LLMs have become powerful tools for generating diverse, complex, and well-structured coding questions efficiently.
• For Problem Solvers (Students/Candidates): The same LLMs undermine assessment integrity by providing instant solutions, creating an uneven playing field.
• Current Limitations: Existing solutions often rely on expensive third-party platforms or lack robust integrity controls (e.g., preventing copy-paste, enforcing full-screen mode). Furthermore, there is a gap between academic practice (often guided learning) and industry hiring (outcome-focused, time-bound pressure testing).
• Resource Constraints: Many educational institutions and startups lack the budget for commercial platforms or the technical infrastructure to host secure, self-contained assessment environments.

2. Methodology

The authors propose OpenCoderRank, a lightweight, self-hosted, and model-agnostic platform designed to bridge the gap between problem setters and solvers.

• Architecture & Stack:
  • Backend: Built using the Python Flask framework.
  • Database: Utilizes SQLite for lightweight data storage, making it easy to deploy in resource-constrained environments.
  • Deployment: Supports Docker for containerization, allowing for rapid, consistent deployment across different systems. It can also be hosted on cloud platforms or run via tunneling tools for minimal overhead.
• Core Workflow:
  1. Question Generation: Problem setters can use an integrated Question Generator that leverages any LLM (via Chain-of-Thought prompting) to create Python, SQL, or MCQ questions. Alternatively, manual creation is supported for non-technical educators.
  2. Assessment Environment: The platform simulates real-world timed technical interviews. It supports both MCQ and Coding modes.
  3. Integrity Controls: To mitigate LLM cheating, the system enforces:
     • Full-screen mode: Participants are forced into full-screen; exiting/disabling it 5 times triggers automatic submission.
     • Copy-Paste Lock: Prevents copying code from external sources (including LLMs).
     • Randomized Sequencing: Questions are shuffled to prevent screen-sharing cheating.
  4. Automated Judging: The system automatically evaluates submissions against predefined input/output test cases for Python and SQL, and checks answers for MCQs.
• Evaluation Strategy:
  • The platform is evaluated based on the quality of generated content, not the specific LLM used.
  • BERTScore: Used to measure semantic similarity between AI-generated questions and expert-authored problems (from LeetCode/HackerRank).
  • LLM-as-a-Judge: A separate state-of-the-art LLM (Gemini 3 Flash) acts as an expert educator to evaluate generated questions on four dimensions: Clarity, Correctness, Pedagogical Usefulness, and Difficulty Alignment.

3. Key Contributions

• OpenCoderRank Platform: A fully functional, open-source (self-hosted) solution for conducting secure, timed technical assessments.
• Model-Agnostic Design: The system does not depend on a specific LLM, allowing institutions to integrate their preferred generation models.
• Integrity-Centric Features: It introduces specific mechanisms (full-screen enforcement, copy-paste blocking, randomization) tailored to the LLM era, which are often missing in free or lightweight educational tools.
• Bridging the Gap: It combines the flexibility of learning platforms with the rigor of industry-style hiring assessments, supporting both academic labs and startup hiring.
• Resource Efficiency: Designed specifically for environments with limited computational resources, requiring minimal configuration for setup.

4. Results

The authors evaluated 100 technical assessment questions generated by the system (using GPT-4o) against expert-authored benchmarks. The results were as follows:

• Semantic Similarity (BERTScore): Achieved an F1 score of 0.79, indicating substantial alignment with expert problems.
• LLM-as-a-Judge Scores (out of 5):
  • Clarity: 4.98 (High readability and unambiguity).
  • Correctness: 4.93 (Technically valid and logically consistent).
  • Pedagogical Usefulness: 4.92 (Well-suited for instruction and assessment).
  • Difficulty Alignment: 4.39 (Generally matched intended difficulty levels).

These results demonstrate that OpenCoderRank can generate high-quality, pedagogically meaningful technical assessments that closely mimic human-crafted problems.

5. Significance

• Democratization of Assessment: By being self-hosted and lightweight, OpenCoderRank allows resource-constrained institutions (universities, coding clubs, startups) to conduct professional-grade technical assessments without expensive vendor contracts.
• Adaptation to the LLM Era: It provides a practical framework for maintaining assessment integrity in a world where AI can solve coding problems instantly, ensuring that evaluations truly measure human problem-solving capabilities.
• Pedagogical Impact: It supports diverse use cases, from intra-college competitions and classroom quizzes to one-on-one mentoring and hiring processes, fostering a more rigorous and fair evaluation ecosystem.
• Future Direction: The paper sets the stage for adaptive learning methodologies, with future work planned to include user profiling, skill trajectory tracking, and more robust anti-cheating verification mechanisms.