💻 computer science

LARC-QL: Q-Learning Enhanced Content Caching for CCN 1 LARC-QL: A Q-Learning Enhanced Latency-Aware and Resource-Efficient Content Caching Scheme for Content-Centric Networks

This paper proposes LARC-QL, a Q-learning enhanced content caching scheme for Content-Centric Networks that utilizes dual Q-tables and real-time demand signals to dynamically optimize caching decisions and on-path placement, significantly improving cache hit ratios, latency, and resource efficiency compared to static strategies.

Original authors: Yasar Khan, Nazia Perwaiz, Saad Mustafa

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yasar Khan, Nazia Perwaiz, Saad Mustafa

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling library system. In the old days (traditional internet), if you wanted a book, you had to ask a specific librarian at a specific desk, and that librarian had to walk all the way to the back room to get it for you, even if a copy was sitting on a shelf just three feet away.

Content-Centric Networking (CCN) changes the rules. Instead of asking for a specific person (an IP address), you ask for the book by its title. Every librarian in the building (every router) has a small shelf (a cache) where they can keep popular books. If a book is on a nearby shelf, you get it instantly. If not, someone has to fetch it from the main vault.

The problem? Librarians are bad at guessing which books will be popular right now. They often keep old, boring books on their shelves because they were popular last year, while missing out on the new bestsellers that everyone is suddenly asking for.

The Old Solution: "LARC"

The researchers previously created a system called LARC. Think of LARC as a very strict, rule-following librarian.

How it works: It has a formula. If a book is requested often, it stays on the shelf. If the shelf is full, it kicks out the book with the lowest score.
The flaw: The formula is static. It's like a librarian who only looks at the total number of times a book has ever been borrowed. It doesn't know if a book is currently trending up or currently dying out. It might keep a book that everyone stopped reading yesterday because it was popular last month, wasting space for the new hit.

The New Solution: "LARC-QL"

This paper introduces LARC-QL, which is like giving that librarian a smart, learning assistant based on a technique called Q-Learning (a type of AI that learns by trial and error).

Here is how LARC-QL works, broken down into simple analogies:

1. The "Trend Spotter" (Dual EWMA)

Imagine the librarian has two sets of eyes:

Fast Eyes: Watching what happened in the last few minutes.
Slow Eyes: Watching what happened over the last hour.

If the "Fast Eyes" see a book being grabbed much more often than the "Slow Eyes" expect, the book is Rising (trending up). If the "Fast Eyes" see fewer grabs than usual, the book is Falling (dying out).

Why it matters: The old LARC system couldn't tell the difference between a book that is always popular and a book that is suddenly popular. LARC-QL spots the trend instantly.

2. The Two "Decision Gates" (Q-Tables)

Instead of just following a formula, LARC-QL uses two tiny, super-fast decision-makers (Q-tables) that learn as they go:

Gate 1 (The "Should We Keep It?" Gate):
- The Question: "Is this book worth putting on the shelf at all?"
- The Learning: If the Trend Spotter says the book is Falling, Gate 1 learns to say, "No, don't waste space on this; let it go." If the book is Rising, it says, "Yes, grab it immediately!"
- The Result: The librarian stops hoarding old, declining content.
Gate 2 (The "Where to Put It?" Gate):
- The Question: "If we keep it, which shelf should it go on?"
- The Learning: The old system tried to put a copy on every shelf the book passed (like photocopying a book and leaving it on every desk in the hallway). This wastes space. Gate 2 learns that the best place to put a popular book is closest to the person asking for it.
- The Result: Fewer copies, but they are in the right places.

3. The "Popularity Bonus"

LARC-QL also gives a special "bonus score" to the absolute most popular books. If a book is a massive hit, the system is extra careful not to throw it away, even if the shelf is crowded. This ensures the "superstars" of the library are always available.

What Did They Find?

The researchers tested this new system in a simulated library network using three different "city layouts" (topologies) and millions of requests. Here is what happened:

More Hits, Less Waiting: The new system found the right book on the local shelf much more often (up to 187% more in some cases). This means users waited less time.
Faster Delivery: On average, getting a book was 4.2 milliseconds faster. In the world of high-speed internet, that's a huge win.
Less Traffic Jams: Because the system stopped wasting space on declining books and placed popular books closer to users, there was less traffic clogging the main roads (links) leading back to the central vault.
Lightweight: The "brain" of this system is tiny. It only needs about 40 kilobytes of memory (less than a single low-resolution photo) to run. It fits easily on standard network equipment without needing expensive new hardware.

The Bottom Line

LARC-QL is like upgrading a librarian from someone who just follows a dusty rulebook to someone who has a real-time radar for what people want right now. It learns to ignore books that are losing popularity and places the hot new books exactly where they are needed most, making the whole internet library faster and less crowded.

Technical Summary: LARC-QL

1. Problem Statement

Content-Centric Networking (CCN) and Named Data Networking (NDN) leverage in-network caching to decouple data delivery from host addressing, aiming to reduce latency and server load. However, designing efficient caching policies remains an open challenge due to four primary factors:

Dynamic Popularity: Content demand follows non-stationary Zipf distributions that change over time.
Resource Constraints: Realistic cache sizes are extremely small (0.04%–0.20% of the content universe).
Lack of Global Coordination: Placement decisions are made locally without global knowledge.
Static Heuristics: Existing schemes, such as the Latency-Aware and Resource-Efficient Content Caching (LARC) algorithm, rely on fixed parameters and cumulative request frequencies. These static approaches fail to distinguish between content that is currently trending upward versus content that is declining in popularity but retains a high cumulative count. Consequently, they often cache declining content and miss opportunities to serve trending content from nearby nodes.

2. Methodology: LARC-QL

The paper proposes LARC-QL, a Q-Learning enhanced extension of the LARC scheme. LARC-QL retains the core Content Access Score (CAS) logic of LARC but replaces two critical static decisions—whether to cache content and where along the path to place it—with online-learned adaptive policies using two independent Q-tables.

Core Architectural Components

Dual Q-Table Architecture:
- $Q_1$ (Miss-Level Gate): Decides whether to attempt caching a specific content item at all. Its state space includes content popularity bins, average path occupancy, path length, and a demand trend signal.
- $Q_2$ (Hop-Level Gate): Determines the optimal node along the return path to store the content. Its state space includes hop distance from the requester, local occupancy, and the demand trend.
- Both tables are trained online using the Bellman equation during simulation without offline training data.
Dual EWMA Demand Trend Detection:
- A novel mechanism calculates a real-time demand trend signal $\tau(C) \in \{0: \text{falling}, 1: \text{stable}, 2: \text{rising}\}$ .
- It utilizes two Exponential Weighted Moving Averages (EWMA): a fast EWMA ( $\theta_f = 0.15$ , tracking ~7 requests) and a slow EWMA ( $\theta_s = 0.02$ , tracking ~50 requests).
- The trend ratio $\rho(C) = \text{EWMA}_f / \text{EWMA}_s$ determines the trend. This signal provides structural information (recency of demand) unavailable to static formulas like LARC's cumulative frequency.
Popularity-Tiered CAS Update:
- The CAS boost factor $\beta$ is weighted by normalized content popularity. High-demand content receives a significantly larger reward per hit (up to 2.6x the base boost), ensuring it retains high CAS values and resists eviction under pressure.
Reward Shaping:
- $Q_1$ Rewards: Correctly skipping declining-trend content yields positive rewards; incorrectly skipping rising-trend content incurs large penalties.
- $Q_2$ Rewards: Includes a proximity reward ( $6/h$ ), encouraging placement closer to the requester (lower hop count) for popular and trending content.
- Hit Rewards: Both tables receive rewards proportional to the latency saved by a cache hit.

3. Key Contributions

Adaptive Decision Making: Replaces static admission and placement thresholds with online Q-learning agents that adapt to real-time network conditions and demand dynamics.
Trend-Aware Caching: Introduces a dual-EWMA mechanism that explicitly detects rising, stable, and falling demand trends, allowing the system to aggressively cache trending content and skip declining content.
Lightweight Implementation: The solution uses a compact state space ( $Q_1$ : 2,400 entries; $Q_2$ : 120 entries) and $O(1)$ lookup costs, making it suitable for resource-constrained ICN routers.
Comprehensive Evaluation: The scheme is evaluated across three diverse topologies (GARR, GEANT, RocketFuel), three Zipf skewness parameters ( $\alpha \in \{0.6, 0.8, 1.0\}$ ), and five cache sizes (0.04%–0.20%).

4. Experimental Results

Simulations were conducted with 900,000 total requests per run (300k warmup, 600k measurement). Key findings include:

Cache Hit Ratio (CHR): LARC-QL achieves up to 187% relative improvement over LARC. The most significant gains occur at $\alpha=0.8$ (standard benchmark), where LARC-QL shows 81–131% relative improvement. At $\alpha=0.6$ (flat distribution), LARC fails completely (CHR=0), while LARC-QL achieves non-zero hits by identifying rising trends.
Latency: Average retrieval latency is reduced by 1.0 to 4.2 ms. The largest reduction (4.2 ms) occurs on the GEANT topology at $\alpha=0.8$ .
Path Stretch: Reduced by 1.6% to 4.6%, indicating content is found closer to the requester due to $Q_2$ 's proximity-based learning.
Link Load: External link load (traffic to the origin server) is reduced by 8–13%, directly correlating with improved CHR. Internal link load is also reduced by 2–3%.
Ablation Study: The study confirms that the combination of the EWMA trend detector and the $Q_2$ hop-level gate accounts for over 85% of the total performance gain. The trend detector alone improves CHR by 15%, while the hop-level gate adds another significant margin.

5. Significance and Claims

The paper claims that LARC-QL represents a significant step forward by integrating genuine online adaptive decision-making into CCN caching. Its primary significance lies in the dual EWMA trend signal, which provides runtime information structurally impossible to capture in static formulas. This allows the system to distinguish between "old popular" content and "new trending" content, a distinction static heuristics cannot make.

The authors emphasize that LARC-QL is immediately deployable on resource-constrained ICN routers without hardware modifications. The computational overhead is less than 5% per request, and the memory footprint is negligible (~41 KB). The paper concludes that the proposed scheme effectively addresses the limitations of static heuristics in dynamic environments, offering a lightweight, model-free reinforcement learning approach that outperforms state-of-the-art baselines across diverse network topologies and content distributions.