Adaptive RAN Slicing Control via Reward-Free Self-Finetuning Agents

This paper proposes a novel self-finetuning framework that enables Generative AI agents to autonomously learn continuous control for dynamic Radio Access Network slicing by distilling long-horizon experiences into model parameters via a bi-perspective reflection mechanism, thereby outperforming traditional Reinforcement Learning and standard LLM-based agents in sample efficiency and multi-objective optimization without relying on handcrafted reward signals.

The Gist

Imagine you are the manager of a busy, high-speed highway (the Radio Access Network or RAN). Your job is to constantly decide how to divide the road lanes (spectrum) among different types of drivers: some are racing cars needing speed (video calls), some are delivery trucks needing steady flow (file downloads), and some are emergency vehicles needing instant access.

The traffic is chaotic and changes every second. If you give too much road to the trucks, the race cars crash. If you switch lanes too often, everyone gets confused and slows down. You need a manager who can make perfect decisions instantly, forever, without getting tired or confused.

The Problem: The "Old Way" vs. The "New Way"

1. The Old Way (Traditional AI/Reinforcement Learning):
Imagine hiring a robot manager. To teach it, you have to write a very strict rulebook (a reward function).

• "If a car waits too long, give the robot a -1 point."
• "If you switch lanes too much, give it a -5 point."
• "If the road is empty, give it a +10 point."

The problem? Writing this rulebook is a nightmare. If the points are slightly off, the robot learns the wrong lesson. It might stop switching lanes entirely to avoid the penalty, even when it's necessary, causing traffic jams. It takes thousands of hours of trial and error to get the math right.

2. The "New" Way (Standard LLM Agents):
Now, imagine hiring a brilliant human expert (a Large Language Model or LLM) who has read every traffic manual in the world. You don't need a rulebook; you just talk to them.

• "Hey, the traffic is heavy, what should we do?"

But there's a catch: This human has a very short memory. They can only remember the last 5 minutes of conversation. If the traffic jam happened an hour ago, they've forgotten it. They also tend to "hallucinate" (make up facts) when the situation gets too complex. They can't learn from their mistakes over the long term because they can't hold the whole story in their head at once.

The Solution: The "Self-Finetuning" Agent

This paper proposes a third option: A Self-Improving Agent that learns like a genius student who writes a diary and then rewrites their own textbook.

Here is how it works, using a simple analogy:

Step 1: The Actor and the Reflector (The Student and the Coach)

Instead of one person doing everything, we have two roles working together:

• The Actor (The Student): This is the AI making the decisions in real-time. It drives the car, switches the lanes, and talks to the traffic system.
• The Reflector (The Coach): This is a smarter version of the AI that watches the entire drive after it's finished. It doesn't just look at the last 5 minutes; it looks at the whole hour.

Step 2: The "Bi-Perspective" Reflection

After a drive, the Coach reviews the Student's diary.

• The Student says: "I switched lanes because the truck was slow."
• The Coach says: "Actually, looking at the whole hour, switching lanes there caused a ripple effect that slowed down the race cars for 10 minutes. That was a bad move. Next time, wait 2 seconds."

The Coach doesn't give a number score (like -5 points). Instead, it writes a linguistic critique: "You were too hasty here. Patience would have been better."

Step 3: The "Self-Finetuning" (Rewriting the Textbook)

This is the magic part. Usually, an AI just reads the Coach's notes and tries to remember them for the next drive. But because the AI has a short memory, it forgets the notes eventually.

Instead, this system rewrites the Student's brain.

1. It takes the Coach's notes and the Student's actions.
2. It creates a "preference dataset" (a list of "Good Moves" vs. "Bad Moves").
3. It uses a special training method (called KTO) to update the Student's internal parameters.

Think of it like this: Instead of the student reading a book of advice, the student absorbs the advice into their muscle memory. The "lesson" becomes part of who they are. They don't need to look at the notes anymore; they just know to wait 2 seconds.

Why This is a Big Deal

1. No Rulebook Needed: You don't need to be a math genius to write complex reward formulas. The AI figures out what "good" looks like by reflecting on its own mistakes.
2. Infinite Memory (Sort of): Even though the AI has a short-term memory limit, it "digests" long-term experiences into its brain weights. It learns from a 10-hour drive and carries that wisdom forward, even though it can't "remember" the whole drive in its head.
3. Super Efficient: In the experiments, this method learned a perfect traffic management strategy using one single drive (one trajectory). Traditional AI needed thousands of drives to get close to this level of performance.

The Result

In the test (managing a 6G network), this new agent:

• Used the road better (higher speed/efficiency).
• Switched lanes less often (more stable, less chaos).
• Kept everyone happy (fewer dropped calls).

It did all this without a human writing a complex rulebook, simply by letting the AI talk to itself, reflect on its mistakes, and permanently upgrade its own "brain" to be smarter next time.

In short: It's the difference between a driver who reads a map and gets lost, and a driver who drives the route once, learns the turns, and then drives it perfectly forever without looking at the map again.

Technical Summary

Here is a detailed technical summary of the paper "Adaptive RAN Slicing Control via Reward-Free Self-Finetuning Agents."

1. Problem Statement

The paper addresses the challenge of achieving autonomous and adaptive control in 6G AI-native networks, specifically focusing on Radio Access Network (RAN) slicing. RAN slicing is a complex, multi-objective optimization problem requiring continuous decision-making to balance:

• Spectrum Efficiency (SE): Maximizing resource utilization.
• Service Quality (QoS): Minimizing packet latency and violation rates.
• Reconfiguration Stability: Minimizing the overhead caused by frequent resource reallocations.

Limitations of Existing Approaches:

• Traditional Reinforcement Learning (RL): While effective, RL suffers from the "reward engineering bottleneck." Designing a scalar reward function that correctly balances conflicting objectives (e.g., throughput vs. stability) requires laborious manual tuning and often fails to generalize across diverse network conditions.
• Large Language Model (LLM) Agents: While LLMs possess strong reasoning capabilities, current agent frameworks (e.g., Reflexion) rely on prompt-based memory (feeding interaction history into the context window). This approach is limited by:
  • Finite Context Windows: Agents cannot retain long-term history.
  • Long Context Degradation: Performance drops as the context grows.
  • Lack of True Continual Learning: Agents cannot internalize experience into their parameters, making them unsuitable for persistent, continuous control tasks.

2. Methodology

The authors propose a Self-Finetuning Framework that enables LLM agents to learn continuously by distilling experience into model parameters rather than relying on expanding prompts. The core components are:

A. Reflective Markov Decision Process (R-MDP)

The authors reformulate the standard MDP to suit LLMs. Instead of scalar rewards, the interaction tuple includes:

• Step-level Reflections ($\psi_t$): Natural language analysis of the previous step.
• Step-level Analyses ($\phi_t$): Justification for the current decision.
• Environment Feedback ($M_t$): Task-specific metrics (e.g., latency, throughput) recorded as part of the trajectory, not used as immediate scalar rewards.

B. Actor-Reflector (AR) Framework

Inspired by the Actor-Critic architecture but adapted for language models:

1. Actor (LLM Policy): Generates actions, step-level reflections, and analyses based on the current state and history.
2. Reflector (Evaluator): A separate LLM module that performs trajectory-level reflection. After a trajectory is completed, the Reflector reviews the entire history, assigns quality labels (True/False) to actions, and proposes improved actions ($\hat{a}_t$) for suboptimal steps. This replaces the scalar value function of a Critic with semantic, language-based feedback.

C. Refine-from-Reflection (RfR) Fine-Tuning

This is the core learning mechanism that converts reflection into parameter updates:

1. Data Construction:
   • Reflector-Labeled Examples: Directly use the Reflector's labels to create positive (effective) and negative (suboptimal) samples.
   • Refine-Rollout Examples: For suboptimal steps, the Actor is sampled multiple times to generate alternative actions. If a sampled action matches the Reflector's suggestion, it is treated as a positive sample. This enhances sample efficiency without new environment interactions.
2. Optimization (KTO): The framework uses Kahneman-Tversky Optimization (KTO) for fine-tuning. Unlike pairwise methods (e.g., DPO), KTO handles unbalanced datasets and models absolute preference likelihoods, allowing the model to internalize the Reflector's preferences directly into its weights.

3. Key Contributions

1. R-MDP & Actor-Reflector Framework: A novel formalism bridging sequential optimization (RL) and semantic reasoning (LLMs), replacing scalar rewards with linguistic feedback.
2. Bi-Perspective Reflection: A mechanism combining step-level in-context learning (Actor) with trajectory-level retrospective analysis (Reflector) to enable dynamic policy adjustment without handcrafted rewards.
3. Refine-from-Reflection (RfR): A self-finetuning pipeline that distills long-horizon experiences into model parameters using KTO, effectively overcoming context window limitations.
4. Reward-Free Continuous Control: Demonstrates that agents can learn complex multi-objective control tasks purely through self-reflection and preference learning, bypassing the need for manual reward engineering.

4. Experimental Results

The framework was evaluated on a dynamic RAN slicing simulator (6G scenario) involving stochastic traffic and frequency-selective fading.

Baselines:

• RL Algorithms: DQN, SAC, PPO (using Ray RLlib).
• LLM Agent: Reflexion (using Qwen3-4B and DeepSeek-R1).

Key Findings:

• Sample Efficiency: The Self-Finetuning agent achieved superior performance with only one training iteration and a single trajectory collection, whereas RL baselines required 80 rounds (1,600 trajectories) and still showed instability.
• Multi-Objective Performance:
  • Spectrum Efficiency (SE): Achieved 5.354 (outperforming Reflexion and DQN; slightly lower than SAC but with better stability).
  • Reconfiguration Times: Achieved 21.091, a 59% reduction compared to PPO and 28.4% lower than Reflexion, indicating significantly higher stability.
  • PQoS Violations: Comparable to Reflexion and significantly better than SAC/DQN.
• Training Dynamics: Analysis showed that the KTO iterations effectively converged, with the model internalizing actionable information from the single trajectory. The "Refine-Rollout" mechanism allowed the agent to explore alternative behaviors without further environment interaction.

5. Significance

• Paradigm Shift: Moves AI-native network control from "prompt-based memory" (which degrades over time) to "parameter-based memory" (true continual learning).
• Solves the Reward Bottleneck: Eliminates the need for domain experts to manually craft complex, multi-objective reward functions, making AI deployment in dynamic networks more scalable.
• Robustness: Demonstrates that generative agents can handle the volatility of 6G networks (traffic fluctuations, channel fading) more effectively than traditional RL or static LLM agents.
• Future Outlook: While current LLM inference speeds limit real-time deployment, the paper suggests that the learned policies can be distilled into lightweight models or optimized via quantization for practical 6G implementation.

In summary, this paper presents a breakthrough in AI-native networking by proving that LLM agents can autonomously master continuous, multi-objective control tasks through self-reflection and parameter distillation, offering a scalable alternative to traditional Reinforcement Learning.