Key Distance Effects in a Controlled Synthetic Rubric-Routing Unit Test for Small Character-Level Transformers

Technical Summary: Key Distance Effects in a Controlled Synthetic Rubric-Routing Unit Test

Problem Statement
Automated educational systems increasingly rely on routing decisions based on structured key fields (e.g., course identifiers, rubric versions, evidence profiles). However, it is difficult to determine whether a model has learned the intended logical lookup or merely exploited surface cues, class imbalances, or single-field predictors. Existing end-task accuracy metrics often conceal the specific mechanisms of failure, such as the inability to retain a specific key, the failure to combine multiple keys, or the difficulty of expressing a mapping through a specific output interface. The paper addresses the need for a diagnostic instrument that isolates the prerequisite "lookup" operation from semantic assessment, allowing researchers to validate routing layers before deployment on authentic text.

Methodology
The study employs a controlled synthetic unit test designed to evaluate small, character-level Transformers trained from scratch.

• Task Design: The experiment creates a fully balanced 4 × 4 mapping between four neutral source keys (representing course/rubric families) and four evidence-profile keys. These 16 pairs are assigned to four neutral action classes. The design ensures that neither the source nor the profile alone predicts the action above the nominal 0.25 chance level, forcing the model to learn the joint association.
• Experimental Conditions: Three conditions test the surface organization of the keys:
  1. Atomic: A single joint key containing both identifiers.
  2. Adjacent Factorized: Two separate keys placed immediately next to each other.
  3. Padded Factorized: Two separate keys separated by 256 neutral space characters.
• Model Architecture: The study uses a character-level decoder-only Transformer with a fixed key-span classifier. The classifier receives mean-pooled hidden states from marked source and profile spans, concatenated with explicit pairwise features (element-wise product and absolute difference). This "head" is purpose-built to facilitate the combination of representations, ensuring that success reflects the backbone's ability to provide informative span representations rather than spontaneous factor discovery.
• Scale and Protocol: The study compares two model sizes: a primary 2.67M parameter model (6 layers, 192 embedding dim) and a 0.8M parameter replication (4 layers, 128 embedding dim). Across 144 frozen runs (varying mapping realizations and initialization seeds), the models were trained to a fixed horizon of 2,000 classification equivalents.
• Endpoints: Primary endpoints include exact four-class accuracy and normalized learning-curve area (AUC). Intervention analysis involves replacing one key within a case to measure shifts in the predicted log-odds toward the action prescribed by the replacement key.

Key Contributions

1. Synthetic Diagnostic Instrument: The paper provides a transparent synthetic test for prerequisite routing operations with explicit marginal nulls and a narrow claim scope, separating factorization capabilities from key spacing effects.
2. Positive-Control Gate: It establishes a "positive-control" regime where a model must first demonstrate learnability on an atomic joint key before claims about factorization or spacing can be made.
3. Isolation of Variables: By holding the mapping, training budget, output head, and query template constant, the study isolates the effects of key factorization and surface distance.
4. Behavioral Intervention Metrics: Instead of relying on aggregate donor accuracy (which is uninformative under balanced permutations), the study uses within-case log-odds shifts to verify that specific keys behaviorally govern the decision.

Results

• Positive Control: The atomic joint-key condition passed as a positive control in both model sizes (mean accuracy ~0.89 for the primary model, ~0.83 for the replication), confirming the interface is learnable.
• Factorization (Adjacent vs. Atomic): In the primary (larger) model, adjacent factorized keys were non-inferior to atomic keys (difference of -0.010, passing the -0.10 margin). In the smaller model, while the mean difference was negative (-0.081), the result was not statistically non-inferior due to higher seed-level variability, suggesting initialization sensitivity rather than a definitive structural failure.
• Key Distance (Padded vs. Adjacent): Separating keys by 256 characters significantly degraded performance in both models.
  • Accuracy: Reduced final accuracy by approximately 0.25 (e.g., from 0.88 to 0.63 in the primary model).
  • Learning Dynamics: Reduced the normalized learning-curve AUC by ~0.04, indicating slower or less sustained formation of the mapping.
  • Confidence: The padded condition exhibited higher prediction entropy and worse expected rank.
• Intervention Analysis: Replacing either the source or profile key in the adjacent condition consistently shifted the predicted distribution toward the action prescribed by the replacement key, confirming that both keys are behaviorally used in the decision process.

Significance and Claims
The paper claims that in this controlled interface, surface key distance, rather than the content of the mapping, is the governing factor for routing success in small models. The findings suggest that while small Transformers can handle factorized keys if they are adjacent, introducing significant spacing (even within the context window) imposes a substantial penalty on both final accuracy and learning speed.

The study emphasizes that this paradigm isolates a lookup mechanism rather than semantic assessment. It argues that before attributing routing failures in educational systems to reasoning deficits or factorization limits, researchers must first establish that the system can express the target mapping through the chosen output channel (positive control) and that the prompt geometry does not artificially hinder the combination of necessary fields. The work motivates the use of synthetic unit tests as a diagnostic step prior to evaluation on authentic text and operational architectures.