Technical Summary: Environmental Feedback-based Credit Assignment (EFCA)

1. Problem Statement

Agentic Reinforcement Learning (RL) utilizing Large Language Models (LLMs) faces a critical challenge in long-horizon interactive environments: sparse and delayed rewards. Unlike conventional RL with verifiable rewards (RLVR) that evaluates single answers or reasoning paths, agentic RL involves sequences of heterogeneous actions where contributions are highly uneven. The final task outcome (success or failure) serves as the primary supervision signal, creating a mismatch between step-level decision-making and episode-level supervision.

Existing solutions attempt to address this through:

1. Trajectory Decomposition: Inferring step-level signals from final outcomes, which often struggles to identify locally effective intermediate actions.
2. Learned Critics: Using parameterized state-value or reward models (e.g., in PPO, VAPO). However, these require training separate evaluators for each environment and can introduce mismatches between the critic's estimates and true environment dynamics.

Crucially, existing methods largely ignore the rich process information naturally generated during environment interaction (e.g., textual feedback, action validity, state changes). This paper argues that such environment-grounded signals provide valuable, underexplored supervision for identifying the contribution of individual actions without requiring external annotation or learned evaluators.

2. Methodology: Environmental Feedback-based Credit Assignment (EFCA)

EFCA is a multi-timescale credit assignment approach designed to refine step-level returns using signals directly extracted from environment feedback. It operates as a plug-and-play module that complements the long-term outcome signal with two environment-grounded process signals, integrated via a return reweighting mechanism. It does not replace the underlying optimizer or introduce an explicit value critic.

2.1 Core Components

The method constructs a unified credit score ($C_t$) at each step $t$ by combining two signals:

1. Short-Term Feedback Signal ($c^{fb}_t$):

   • Goal: Captures the immediate effect of the current action.
   • Mechanism: Analyzes the textual feedback ($f_t$) returned by the environment. Using predefined positive ($P$) and negative ($N$) pattern sets (e.g., "You pick up..." vs. "Nothing happens"), the system assigns a credit of $+1$ (progress), $-1$ (failure/invalid), or $0$ (neutral).
   • Function: Determines if the current action produced a meaningful local state transition.

2. Medium-Term State-History Signal ($c^{hist}_t$):

   • Goal: Identifies ineffective patterns and short-horizon behavioral loops.
   • Mechanism: Examines a recent window ($K$) of feedback credits. It assigns a penalty of $-1$ if the window contains consecutive negative feedback (repeated failures) or a mild penalty ($-\eta$) if the window contains no positive feedback (stagnation).
   • Function: Detects and suppresses repeated ineffective behaviors or lack of progress over a short horizon.

2.2 Return Reweighting

The combined credit score is transformed into a step-dependent weight ($w_t$) to modulate the original base return ($R^{base}_t$), which is typically computed via discounted accumulation of trajectory rewards.

• Credit Score: $C_t = \alpha \cdot c^{fb}_t + \beta \cdot c^{hist}_t$
• Weight Calculation: $w_t = \text{clip}(1 + \lambda C_t, w_{min}, w_{max})$
• Final Reweighted Return: $\tilde{R}_t = w_t \cdot R^{base}_t$

This mechanism adaptively calibrates supervision: it strengthens the signal for locally useful actions and suppresses signals for redundant or ineffective steps, all while preserving compatibility with existing stepwise optimization pipelines (e.g., GiGPO, HGPO).

3. Key Contributions

• Novel Signal Source: The paper proposes leveraging naturally occurring environment feedback (textual responses, validity checks) as a primary source for step-level credit assignment, bypassing the need for learned reward models or external annotations.
• Multi-Timescale Framework: It introduces a structured approach to credit assignment that integrates:
  • Long-term outcome signals (final rewards).
  • Short-term execution signals (immediate feedback).
  • Medium-term efficiency signals (recent history of stagnation/failure).
• Lightweight Implementation: EFCA is designed as a lightweight reweighting module that requires no auxiliary value networks, making it computationally efficient and easily integrable with existing critic-free stepwise optimization methods.
• Empirical Validation: The method is validated on two representative long-horizon benchmarks: ALFWorld (embodied text-based decision making) and WebShop (web-based shopping).

4. Experimental Results

Experiments were conducted using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct backbones.

• Performance Gains: EFCA consistently outperformed strong baselines, including prompting-based methods (ReAct, Reflexion) and RL-based stepwise optimizers (PPO, GRPO, GiGPO, GraphGPO, HGPO).
  • On ALFWorld, EFCA achieved the highest overall scores for both model scales (e.g., 96.03 with Qwen2.5-7B, surpassing HGPO by +4.04 points).
  • On WebShop, EFCA achieved the best Task Score (measuring semantic alignment with user instructions regarding attributes, options, and price), reaching 89.81 with the 1.5B model and 89.06 with the 7B model. Notably, while GraphGPO had a slightly higher task success rate, EFCA's superior Task Score indicated better alignment with the specific product requirements.
• Training Dynamics:
  • Reweighting Ratio: The mean reweighting ratio increased from near 1.0 in early training to ~1.4–1.5 in later stages, indicating the policy increasingly produced actions supported by positive process evidence.
  • Variance: The standard deviation of reweighted returns was higher than base returns, demonstrating that EFCA successfully differentiates between high-contribution and low-contribution steps rather than applying uniform scaling.
• Ablation Studies: Removing either the short-term feedback or medium-term state-history signal resulted in significant performance drops, confirming the complementary nature of both signals.

5. Significance and Claims

The paper claims that EFCA offers a practical and effective complement to existing group-based RL methods for long-horizon agentic tasks. Its primary significance lies in demonstrating that:

1. Environment-grounded signals are sufficient: High-quality step-level supervision can be derived directly from the interaction process without training separate reward models.
2. Multi-timescale refinement is crucial: Combining immediate feedback with historical context effectively mitigates the sparsity and delay of rewards, guiding the policy toward locally effective actions and preventing stagnation.
3. Scalability: The method improves performance across different model scales (1.5B to 7B) and diverse environments (embodied agents and web agents) with negligible computational overhead compared to the backbone LLM training.

The authors position EFCA not as a replacement for advanced reward modeling, but as a streamlined, interpretable, and immediately applicable strategy to enhance credit assignment in scenarios where environment feedback is rich but final rewards are sparse.