Q-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis

Technical Summary: Q-Learning Lab

Problem Statement
Reinforcement learning (RL) is increasingly central to undergraduate curricula, yet the introduction of tabular Q-learning often relies on abstract presentations of the Bellman update equation. Students typically observe policy convergence (e.g., colored grid cells and arrows) without witnessing the mechanistic details of value computation or action selection. Existing teaching tools, ranging from the Berkeley CS188 suite to modern browser-based playgrounds, visualize the outcome of learning but keep the update process hidden within code. This opacity prevents learners from understanding specific mechanistic questions, such as which Q-values were inspected before a move, whether an action was exploratory or greedy, or how specific state transitions (like wall collisions) influence the temporal-difference (TD) error.

Methodology
The authors present Q-Learning Lab, a single-file, browser-based, bilingual (Thai/English) tool designed to make the Q-learning algorithm fully inspectable. The system operates on a deterministic 5×5 gridworld with a start state, a goal, pits, and walls. The methodology is defined by three core design features:

1. Live Bellman Substitution: A dedicated panel renders the Bellman update equation at every step with actual numeric values substituted for variables ($r$, $\gamma$, $Q(s', a')$, etc.). This explicitly shows the TD error, the bootstrap term, and how terminal states or wall collisions (where $s' = s$) alter the update.
2. Decision-Complete Trace Logging: Every transition is logged with fourteen fields, including the full pre-action Q-row, the greedy argmax, the exploration flag (indicating if the action was random or greedy under $\epsilon$-greedy), the TD target, and pre/post-update Q-values. This log functions as a human-readable replay buffer.
3. Learn–Export–Analyze Loop: Learners can export the complete trace as a CSV file. This enables students to generate their own datasets to plot learning curves, value heatmaps, and visitation maps, transforming the learning process from passive observation into reflective inquiry based on learner-generated data.

The tool is implemented as a single HTML file (~600 lines) with no external dependencies, ensuring it runs offline in any modern browser.

Key Contributions

• Epistemic Shift: The paper proposes a shift in the learner's relationship with the algorithm from "spectator of a converging process" to "analyst of a recorded process."
• Pedagogical Instrument: The tool operationalizes constructionist and experiential learning theories by providing a tangible artifact (the trace dataset) for students to build, inspect, and analyze.
• Technical Validation without Human Subjects: Instead of a user study, the paper validates the tool through three technical evaluations:
  1. Correctness: The learned values and policies are compared against a value-iteration ground truth on the identical MDP.
  2. Reproducibility: Hyperparameter sweeps ($\alpha$, $\gamma$, $\epsilon$) are conducted to verify that pedagogical claims (e.g., the effect of myopic discounting) are reproducible properties of the algorithm.
  3. Diagnostic Capability: A reward-editing study demonstrates the tool's ability to distinguish between two behaviorally identical but diagnostically opposite failure modes: exploration failure (premature convergence) versus genuine reward misspecification.

Results

• Algorithmic Correctness: Against a value-iteration ground truth ($V^*$), the Q-learning implementation achieved an average agreement of 98.5% on non-terminal states across 10 random seeds. The maximum absolute error in value estimation averaged 0.34, with residual errors confined to rarely visited off-path states, accurately reflecting the off-policy nature of the algorithm.
• Hyperparameter Reproducibility: Experiments confirmed that $\gamma=0$ fails to solve the task (myopic behavior), while $\epsilon=1$ (pure exploration) still converges to the optimal policy, demonstrating the off-policy learning capability of Q-learning.
• Diagnostic Distinction: The reward-editing study successfully differentiated two regimes where the agent ends in a pit:
  • Regime A ($R_{pit}=+5$): The agent ends in a pit due to exploration failure (premature convergence to a local optimum), not because the pit is optimal. The value-iteration ground truth confirms the goal path remains superior.
  • Regime B ($R_{pit}=+20$): The agent ends in a pit because the reward specification makes the pit genuinely optimal. The learned policy matches the ground-truth optimal policy for the edited MDP.
    This distinction highlights that diagnosing RL agents requires comparing behavior against a computed optimum, not just observing behavior.

Significance and Claims
The paper claims that Q-Learning Lab reframes RL education from an animation to be watched into an instrument that produces data to be analyzed. The significance lies in the tool's ability to make the "black box" of the Bellman update transparent and to provide a mechanism for learners to engage in reflective analysis of their own experiments.

The authors explicitly state that the paper's goal is technical and pedagogical design validation, not the measurement of learning gains. They argue that technical verification of the tool's claims is a prerequisite for future classroom studies. The tool is positioned not as a competitor to advanced multi-agent research infrastructure, but as a foundational preparation that teaches the mechanics of updates, exploration, and reward authorship before students encounter the complexity of multi-agent systems. The authors remain modest, noting that the tool is limited to tabular Q-learning on a deterministic gridworld and that future work will involve classroom evaluation with human subjects.