Technical Summary: ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate

1. Problem Statement

Reinforcement Learning (RL) for Large Language Models (LLMs) faces a persistent challenge in credit assignment: distributing a sparse, trajectory-level reward (e.g., correctness of a final answer) across hundreds or thousands of token-level decisions. Recent literature has proposed various intrinsic token-level weighting schemes—such as surprisal, entropy reduction, and policy divergence—to redistribute this credit more effectively than uniform allocation.

Simultaneously, practical LLM-RL pipelines increasingly rely on Parameter-Efficient Fine-Tuning (PEFT), specifically LoRA (Low-Rank Adaptation), due to memory and compute constraints. However, these two research threads have developed in isolation. This paper argues that this separation creates a structural failure mode: intrinsic credit signals derived from output distributions (log-probabilities) degenerate under the geometric constraints of LoRA.

Under LoRA, the policy is restricted to a low-rank neighborhood of a frozen reference model. Consequently, the per-token differences in output distributions (surprisal, entropy, divergence) that standard weighting schemes rely on become either:

1. Uniformly broadcast: The signal becomes flat because the LoRA-induced perturbations are too small to overcome the base model's dominant uncertainty.
2. Spuriously sparse: Normalization artifacts concentrate credit on a tiny, task-agnostic set of positions.

In both cases, the weighting signal fails to reflect where the adapter actually learns to act, rendering the credit assignment mechanism ineffective before training even begins.

2. Methodology

2.1. Diagnosis of Degeneration

The authors formalize the degeneration of intrinsic signals under low-rank adaptation. They define the policy $\pi_\theta$ as a perturbation of a reference policy $\pi_{ref}$ via a LoRA adapter.

• Surprisal and Entropy: The perturbation to log-probabilities is bounded by the adapter's rank and norms. In a well-trained base model, the first-order perturbation is dominated by the base model's existing uncertainty pattern rather than the adapter's specific updates.
• Policy Divergence: The divergence $|\log \pi_\theta - \log \pi_{ref}|$ is constrained by the KL budget imposed by the low-rank adapter. This forces per-token log-probability ratios to be small and approximately uniform.
• Consequence: When these scores are normalized within a trajectory (as required for gradient updates), the resulting weights either approach a uniform distribution ($Gini \approx 0$) or collapse into a spurious sparse distribution ($Gini \approx 1$, low Effective Token Ratio), failing to track the adapter's actual learning dynamics.

2.2. Proposed Solution: ARCA

To bypass this failure mode, the authors propose Adapter-Residual Credit Assignment (ARCA). Instead of measuring properties of the output distribution (which are flattened by LoRA), ARCA measures the adapter's direct contribution to the hidden states.

• Definition: For a token at position $t$, the salience score $\alpha^{ARCA}_t$ is defined as the $L_2$ norm of the adapter residual:
  $$\alpha^{ARCA}_t(x, y) = \|h^{adapted}_t - h^{base}_t\|_2$$
  where $h^{adapted}_t$ is the hidden state with the adapter enabled, and $h^{base}_t$ is the hidden state with the adapter disabled.
• Implementation: This requires a single extra forward pass per minibatch with the adapter disabled (a standard operation in PEFT frameworks like PEFT to compute KL regularizers). No learned value heads, reward models, or tree constructions are required.
• Theoretical Justification: While the adapter's impact on the output logits is constrained to be small and uniform, its impact on hidden state activations retains position-to-position variation. This is because the adapter's input activations (routing through attention and MLPs) are non-uniform across the sequence (distinguishing content from filler). Thus, the norm of the residual $\|\Delta h_t\|$ provides a non-degenerate, adapter-aware signal of where the model is actually changing.

3. Key Contributions

1. Identification of a PEFT-Specific Failure Mode: The paper formalizes how output-distribution-based credit signals (surprisal, entropy, divergence) degenerate under LoRA, leading to uniform or spuriously sparse token weights that do not reflect the adapter's learning.
2. Introduction of ARCA: A lightweight, adaptation-aware credit assignment rule that derives token salience from the adapter's hidden-state residual. It requires no additional learnable parameters beyond the LoRA adapter itself.
3. Empirical Validation: The authors provide concentration diagnostics (Gini coefficient and Effective Token Ratio) and performance benchmarks on the MATH dataset using Qwen3-1.7B. They demonstrate that ARCA maintains a non-degenerate credit distribution in the low-rank regime where baselines fail, while remaining competitive in downstream performance.

4. Experimental Results

4.1. Setup

• Model/Task: Qwen3-1.7B-Base fine-tuned on MATH using Group Relative Policy Optimization (GRPO).
• Baselines: Uniform weighting, Surprisal, Entropy Reduction, and Policy Divergence (all with LoRA rank $r=64$).
• ARCA Variants: Tested at ranks $r=4, 16, 64$.
• Metrics: Held-out greedy accuracy, pass@4, and token-weight concentration diagnostics (Gini, Effective Token Ratio).

4.2. Diagnostic Results (Credit Assignment)

• Baselines: Surprisal, Entropy, and Divergence weighting exhibited degenerate behavior under LoRA. Their effective token ratios were extremely low (0.052–0.073), indicating spurious sparsity where credit was concentrated on a tiny fraction of tokens, and Gini coefficients were near 1.
• ARCA: Occupied a "middle regime." At rank 64, ARCA achieved a Gini of 0.490 and an effective token ratio of 0.334. This confirms the theoretical claim: ARCA produces a non-uniform signal that is not degenerate into spurious sparsity, effectively tracking where the adapter acts.

4.3. Performance Results

• Rank-Matched Comparison (r=64): ARCA (r=64) was competitive with the baselines.
  • Greedy Accuracy: ARCA (0.590) trailed Uniform (0.620) but outperformed Surprisal (0.520) and Policy Divergence (0.565).
  • Pass@4: ARCA (0.680) was competitive with Entropy Reduction (0.700) and Uniform (0.675).
• Low-Rank Controls: ARCA at lower ranks (r=4, r=16) did not outperform the rank-64 baselines, consistent with a capacity-performance tradeoff rather than an artifact of extra parameters.

5. Significance and Claims

The paper modestly claims to provide a mechanistic explanation for the instability and failure of full-token GRPO + LoRA observed in prior work (e.g., Lee & Tong, 2025). It argues that the failure is not in the RL objective itself, but in the interaction between the adaptation strategy (LoRA) and the credit assignment signal.

The significance of ARCA lies in its ability to:

1. Decouple Credit Assignment from Output Distribution: By measuring hidden-state residuals, ARCA avoids the geometric constraints that flatten output-distribution signals in LoRA.
2. Provide a Lightweight Alternative: It requires no learned critics, value heads, or complex tree structures, making it compatible with existing critic-free RLVR pipelines.
3. Restore Non-Degenerate Signals: It ensures that token-level credit assignment remains meaningful and adapter-aware even when the policy is constrained to a low-rank neighborhood.

The authors emphasize that their MATH/Qwen3-1.7B sweep is a targeted validation of the mechanism rather than a broad benchmark claiming universal superiority. The primary contribution is demonstrating that adaptation-aware credit assignment changes the geometry of token updates in the predicted way, offering a viable path forward for token-level credit assignment in parameter-efficient RL.