💻 computer science

Agentic CAMA-DRL: A Context-Aware Multi-Agent Deep Reinforcement Learning Framework for Multi-Stakeholder Charging Coordination of Last-Mile Delivery E-Bikes

This paper proposes Agentic CAMA-DRL, a context-aware multi-agent deep reinforcement learning framework that coordinates delivery operators, charging stations, fleet planners, and environmental regulators to optimize e-bike charging in urban last-mile logistics, achieving significant improvements in fleet utilization, punctuality, congestion reduction, and CO2 emissions compared to traditional rule-based approaches.

Original authors: Muddsair Sharif, Huseyin Seker

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muddsair Sharif, Huseyin Seker

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a city as a giant, bustling game of tag where the players are delivery bikes and the "bases" are charging stations. In the past, these bikes ran on diesel, but now, like many of us swapping our phones for newer models, they are switching to electricity. This is great for the air we breathe, but it introduces a tricky new rule: the bikes have to stop and plug in to keep playing. If everyone tries to plug in at the same time, or if a bike stops to charge just when it's supposed to be dropping off a package, the whole game grinds to a halt. This is the world of "last-mile delivery"—the final, often chaotic leg of getting a package from a warehouse to your doorstep. Scientists who study this field use a tool called Deep Reinforcement Learning. Think of this as a super-smart video game AI that learns by playing millions of rounds, trying different moves, and remembering which ones get the highest score. When you have many different players (like the delivery company, the charging station owners, and the city regulators) all trying to win at the same time, it becomes a Multi-Agent problem. The big question researchers are asking is: Can we teach these AI players to work together as a team, rather than fighting each other, while also paying attention to the changing weather, traffic, and electricity prices?

This paper introduces a new, clever system called Agentic CAMA-DRL to solve exactly that puzzle. The authors, Muddsair Sharif and Huseyin Seker, built a digital simulation of a busy morning in London to test their idea. Instead of having one boss tell every bike what to do, they created four different "AI agents," each with a specific job and a different perspective. One agent is the Delivery Operator, whose only goal is to make sure packages arrive on time. Another is the Charging Station Operator, who wants to make sure no station gets too crowded. A third is the Fleet Planner, who tries to save money by charging when electricity is cheap. The fourth is the Environmental Planner, who wants to use green energy from the sun whenever possible.

The magic of this system is how these four agents talk to each other. They don't send text messages back and forth; instead, they all look at the same "context"—a live feed of data that includes traffic jams, how much battery is left in every bike, and when the sun is shining. They all share a common scoreboard. If one agent makes a move that helps them but hurts the others (like sending a bike to a charger that is already full), the whole team's score goes down. This forces them to learn a strategy where everyone wins together. The system is "agentic," meaning it doesn't just react to a low battery; it anticipates problems. It's like a chess player who doesn't just move a piece to save it from capture, but moves it three turns ahead to set up a winning position.

When the researchers ran their simulation, the results were impressive. Compared to the old, simple rules (like "charge whenever the battery is low") or systems that only had one AI boss, this new team-based approach made the fleet work 28% more efficiently. It reduced traffic jams at charging stations by 35% and helped deliveries arrive on time 32% more often. Perhaps most importantly for our planet, it cut carbon dioxide emissions by 42%. The authors also ran a special test to see how much of this success came from the teamwork versus the "context" (the real-time data). They found that while teamwork was huge, the ability to see the future context added an extra 5–8% boost, proving that the two ideas work better together than apart.

It is important to remember that these numbers come from a highly realistic computer simulation of London, not from a real-world test on actual streets yet. The authors are careful to say this is a "deployment-ready" solution, meaning the software is built and tested in the lab, but it still needs to be tried out in the real world to see how it handles unpredictable human riders or broken sensors. However, the study suggests that by treating delivery logistics as a cooperative team sport rather than a solo race, we can make our cities cleaner, faster, and much less frustrating for everyone involved.

Technical Summary: Agentic CAMA-DRL for Last-Mile E-Bike Charging

1. Problem Statement

The rapid electrification of urban last-mile logistics has introduced a critical bottleneck: the coordination of charging for delivery e-bikes. Unlike commuter micro-mobility, delivery operations face strict time-window constraints, variable payload-dependent energy consumption, and multi-depot coordination requirements. Current charging strategies are often reactive or rule-based, leading to suboptimal outcomes such as missed delivery windows, station congestion, and avoidable carbon emissions.

Existing solutions fail to simultaneously address three specific challenges:

Delivery-specific constraints: Time-window adherence and payload-adjusted energy budgets.
Multi-stakeholder conflict: The need to coordinate heterogeneous stakeholders (delivery operators, station operators, fleet planners, and environmental regulators) with potentially conflicting objectives.
Agentic context integration: The lack of systems that can anticipate future conditions (demand, weather, grid carbon intensity) rather than merely reacting to current states.

2. Methodology: Agentic CAMA-DRL

The authors propose Agentic CAMA-DRL (Context-Aware Multi-Agent Deep Reinforcement Learning), a framework designed to optimize charging infrastructure through cooperative, proactive decision-making.

2.1 System Architecture

The framework operates as a continuous closed-loop system comprising four functional columns:

Data Sources: Real-time streams including GPS/trip data, charging station metadata (occupancy, charger type, energy source), demand/weather forecasts, and route constraints.
Agentic Context Engine: A five-stage pipeline that transforms raw data into an augmented state vector ( $\tilde{s}_t$ $\tilde{s}_{t}$ ):
1. State Encoding: Normalization and temporal embedding of time-of-day and day-of-week.
2. Forecasting: Generation of 30–60 minute demand forecasts and State-of-Charge (SoC) trajectories.
3. Feature Fusion: Concatenation of observations and forecasts into a latent representation.
4. Environmental Signal Injection: Overlaying time-of-use tariffs, grid carbon intensity, and solar availability.
5. State Assembly: Broadcasting the augmented context vector to all agents.
CAMA-DRL Agents: Four specialized Deep Q-Network (DQN) agents, each mapped to a distinct stakeholder:
- Delivery Operator: Prioritizes on-time delivery and opportunistic charging windows.
- Station Operator: Balances inbound demand to maintain 60–80% occupancy.
- Fleet Planner: Schedules proactive low-tariff top-ups and manages fleet SoC distribution.
- Environmental Planner: Steers charging toward solar-supplemented stations to minimize CO2.
Stakeholder Outputs & Feedback: Agents receive rewards based on system-wide outcomes (utilization, congestion, punctuality, CO2), closing the loop.

2.2 Coordination Mechanism

The framework utilizes a Centralized Training, Decentralized Execution (CTDE) variant.

Shared Reward Structure: All four agents optimize a common scalarized reward function:
$R_k(t) = w_1 \Delta U_t - w_2 C^{cong}_t + w_3 D^{on-time}_t - w_4 CO_2^t$
Where $\Delta U_t$ is fleet utilization, $C^{cong}_t$ is congestion penalty, $D^{on-time}_t$ is on-time delivery, and $CO_2^t$ is carbon cost.
Implicit Coordination: Agents do not exchange explicit messages. Instead, the shared reward structure ensures that an action benefiting one stakeholder at the expense of another (e.g., causing congestion) reduces the overall reward signal, discouraging such behavior through gradient updates.
Training Algorithm: Agents use independent replay buffers but share a common DQN backbone. Training employs standard DQN stabilizers: experience replay, target networks (updated every 10 episodes), and $\epsilon$ -greedy exploration with geometric decay.

2.3 Training Configuration

Discount Factor ( $\gamma$ ): Set to 0.9 to enable far-sighted, anticipatory planning (long-horizon optimization).
Hyperparameters: Batch size of 128, 3 hidden layers of 256 units, and a reward weight set of $\{0.2, 0.3, 0.4, 0.1\}$ prioritizing on-time delivery and congestion avoidance during peak hours.

3. Key Contributions

The paper identifies three principal contributions:

Novel Architecture (C1): A four-agent DQN architecture operating cooperatively over a shared urban environment with a fully contextualized state representation, enabling proactive rather than reactive decisions.
Multi-Stakeholder Reward (C2): A reward function that balances fleet utilization, punctuality, station occupancy, and CO2 cost without requiring explicit inter-agent communication protocols, enabling Pareto-improving coordination.
Empirical Superiority (C3): Validation in a high-fidelity London simulation showing significant gains over rule-based, single-agent DQN, and context-free multi-agent baselines, with ablation studies isolating the specific value of context encoding.

4. Experimental Results

The framework was evaluated in a high-fidelity simulation of London's morning peak (06:00–10:00) involving 50 e-bikes, 20 charging stations, and stochastic demand.

4.1 Performance Metrics

Compared to a Rule-Based (RB) baseline, the full Agentic CAMA-DRL framework achieved:

Fleet Utilisation Rate (FUR): +28% increase (reaching 97%).
Charging Station Congestion: 35% reduction in peak occupancy (dropping to 40%).
Delivery Punctuality Rate (DPR): +32% improvement (reaching 95%).
CO2 Emissions: 42% reduction (from 0.48 to 0.28 kg/delivery).

4.2 Ablation Analysis

Multi-Agent vs. Single-Agent: Moving from Single-Agent DQN to Multi-Agent coordination improved FUR by 5% and reduced congestion by 10%.
Context-Awareness: Comparing "CAMA-DRL (no context)" against the full model revealed that real-time context encoding (demand forecasts, tariffs, solar availability) contributed an additional 5–8% improvement in performance metrics (specifically FUR, DPR, and CO2). This confirms that context and coordination are synergistic, not merely additive.

4.3 Stakeholder Benefits

Delivery Operators: 27% revenue gain, 16% cost reduction.
Station Operators: 24% cost reduction due to reduced maintenance from congestion.
Fleet Planners: 21% revenue gain, 18% cost reduction via uniform SoC distribution.
Environmental Planners: 32% CO2 reduction driven by solar-priority routing.

5. Significance and Claims

The paper positions Agentic CAMA-DRL as a scalable, deployment-ready solution for sustainable urban last-mile logistics. Its significance lies in:

Synergy of Mechanisms: Demonstrating that combining multi-agent coordination with real-time context encoding yields results superior to the sum of its parts.
Anticipatory vs. Reactive: Establishing that "agentic" behavior (planning across extended horizons) is critical for handling the temporal variability of delivery demand and renewable energy supply.
Practical Deployment: The framework requires no explicit inter-agent communication and executes inference in under 1 millisecond per decision step, making it suitable for real-time control of large fleets.
Modularity: The architecture supports a clear migration path toward mixed fleets (e-bikes and electric vans) sharing common charging hubs.

The authors maintain a modest stance regarding limitations, noting that results are based on simulation (though calibrated to realistic parameters) and that the reported gains are relative to the specific baselines tested. They acknowledge that sim-to-real transfer may face challenges from sensor noise and human rider behavior, and that reward weights require careful tuning to avoid biasing specific stakeholders. Future work is proposed to include federated learning, hierarchical agents for mixed fleets, and live grid integration.