🤖 AI

Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading

The paper proposes "Moira," a language-driven hierarchical reinforcement learning framework that utilizes large language models for both high-level pair selection and low-level execution, optimizing them exclusively through prompt updates based on textual feedback to address credit assignment challenges in pair trading.

Original authors: Polydoros Giannouris, Yuechen Jiang, Lingfei Qian, Yuyan Wang, Xueqing Peng, Jimin Huang, Guojun Xiong, Sophia Ananiadou

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Polydoros Giannouris, Yuechen Jiang, Lingfei Qian, Yuyan Wang, Xueqing Peng, Jimin Huang, Guojun Xiong, Sophia Ananiadou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a very sophisticated trading firm, but instead of hiring a team of humans, you have two specialized AI assistants working together. This paper introduces a system called Moira that uses these two AI assistants to make money by trading pairs of stocks (buying one and selling another simultaneously).

Here is how Moira works, explained through simple analogies:

1. The Problem: The "Blame Game"

In traditional trading systems, if you lose money, it's often a mystery. Did you pick the wrong stocks to trade? Or did you just buy and sell at the wrong times? It's like a chef serving a bad meal and not knowing if the problem was the recipe (the high-level plan) or the cooking (the low-level execution). Because the feedback (profit or loss) comes days or weeks later, it's hard to know who to blame or how to fix it.

2. The Solution: A Two-Manager Team

Moira solves this by splitting the job into two distinct roles, just like a business has a Strategist and a Tactician.

The Strategist (The "Selector"): This is the high-level boss. Their only job is to look at the big picture—economic news, long-term trends, and market sentiment—and pick one pair of stocks to trade for the next month. Once they pick a pair, they step back. They don't touch the buttons.
The Tactician (The "Trader"): This is the hands-on worker. Once the Strategist picks a pair (e.g., Amazon and Meta), the Tactician takes over. Their job is to watch the price movements minute-by-minute and decide exactly when to buy or sell to make a profit.

3. The Secret Sauce: "Talking" to Improve

Usually, AI learns by adjusting complex math equations (gradients). Moira does something different: it learns by talking to itself.

How it works: Both the Strategist and the Tactician are powered by Large Language Models (like the AI behind this explanation). Instead of changing their internal math, the system changes the instructions (prompts) given to them.
The Feedback Loop:
- For the Tactician: After a week of trading, a "Critic" AI looks at the trades made. It writes a note in plain English: "You bought too early when the news was bad. Next time, wait for the news to settle." The system then rewrites the Tactician's instruction manual to include this advice.
- For the Strategist: At the end of the month, the Critic looks at the total profit. It writes a note: "You picked a pair that was too volatile. Next time, look for pairs with more stable news." The system rewrites the Strategist's instruction manual.

4. Why This is Better

The paper tested this against other methods (like old-school math formulas and other AI models) using real US stock market data.

The Result: Moira made significantly more money and took less risk than the others.
The Analogy: Imagine a flat AI trying to do everything at once (picking stocks and timing the trades). It gets confused, like a person trying to write a novel while simultaneously cooking dinner. Moira separates the tasks. The Strategist focuses on the "story" (which stocks), and the Tactician focuses on the "action" (when to trade).
The "Tuning" Effect: The paper found that if you just give the AI a basic instruction without letting it "learn" from its mistakes via these text updates, it performs poorly. But once the AI starts rewriting its own instruction manual based on feedback, it becomes a disciplined, profitable trader.

Summary

Moira is a trading system that uses language to teach AI how to trade. It separates the job of choosing what to trade from the job of executing the trade. By having the AI "critique" its own performance in plain English and update its own instructions, it learns to avoid mistakes and capture profits much better than traditional methods.

Key Takeaway: The paper claims that by separating the "big picture" decision from the "nitty-gritty" execution and using language to update the AI's rules, you can solve complex financial problems where it's usually hard to tell who is at fault for a loss.

Technical Summary: Moira – Language-Driven Hierarchical Reinforcement Learning for Pair Trading

1. Problem Formulation

The paper addresses the challenge of sequential decision-making in environments characterized by hierarchical structure, delayed feedback, and ambiguous credit assignment. Specifically, the authors focus on pair trading, a financial domain that naturally combines two distinct temporal and semantic scales:

High-Level Abstraction (Pair Selection): Requires long-horizon semantic reasoning to identify asset pairs with stable economic or statistical relationships.
Low-Level Execution (Trading): Requires short-horizon, reactive decision-making under partial observability (market noise) to execute market-neutral trades.

The core difficulty lies in credit assignment: when a trading strategy fails, it is unclear whether the error stems from a flawed high-level abstraction (choosing the wrong pair), suboptimal low-level execution (poor timing), or the interaction between the two. Traditional end-to-end reinforcement learning (RL) entangles these components, making it difficult to isolate the source of failure. Furthermore, standard Hierarchical RL (HRL) often relies on rigid temporal abstractions or assumes fixed noise distributions, limiting adaptability when semantic relationships drift.

2. Methodology: Moira Framework

The authors propose Moira, a language-driven hierarchical reinforcement learning framework. Instead of using gradient-based fine-tuning or expert-labeled data, Moira parameterizes both the high-level selector and the low-level trader as prompt-conditioned Large Language Models (LLMs). The policies are optimized exclusively through prompt updates driven by textual feedback.

2.1 Hierarchical Policy Design

The framework explicitly separates abstraction selection from execution, aligning their adaptation schedules with the availability of feedback:

High-Level Selector ( $\pi_{sel}$ ): Operates at the episode level (e.g., monthly). It observes aggregated price statistics and semantic summaries (news, sentiment) to select an asset pair $z$ . This selection remains fixed for the duration of the episode.
Low-Level Trader ( $\pi_{tr}$ ): Operates at the intra-episode level (e.g., daily/weekly). Conditioned on the fixed pair $z$ , it executes trading actions based on real-time spread dynamics and local price features.

2.2 Optimization via Prompt Adaptation

The system utilizes a two-stage textual optimization process for both levels, avoiding gradient descent on model weights:

Trajectory-Level Critique (Stage 1): A "Critic" LLM analyzes the executed trajectory (observations, actions, rewards) and generates a natural language feedback signal ( $g_{text}$ ). This signal identifies flaws in decision rules (e.g., "premature exits" or "ignoring news catalysts").
Prompt Update (Stage 2): An "Update" LLM incorporates the textual feedback into the current policy prompt, generating an updated prompt ( $\theta^{(k)}$ ) for the next sub-episode or episode.

Key Distinction in Adaptation Schedules:

Trader: Undergoes intra-episode prompt updates. It adapts its execution rules based on trajectory-level feedback while the pair context remains fixed.
Selector: Undergoes episodic prompt updates. It receives delayed, aggregated feedback (cumulative return) after the Trader has finished its adaptation and execution for the episode.

3. Key Contributions

Language as a Shared Semantic Interface: The paper introduces a novel approach where language serves as the interface between hierarchical levels. Unlike scalar rewards, textual feedback allows for explicit semantic attribution, distinguishing between abstraction errors (bad pair choice) and execution errors (bad timing).
Prompt-Driven Hierarchical Optimization: The authors demonstrate that LLMs can function as hierarchical policies optimized solely through prompt updates. This induces a natural separation of time scales without requiring complex algorithmic constraints or gradient-based fine-tuning.
Decoupling Abstraction and Execution: By fixing the pair selection for an episode while allowing the trader to adapt intra-episode, the framework reduces non-stationarity across levels and enables targeted adaptation under delayed feedback.

4. Experimental Results

The framework was evaluated on historical U.S. equity market data (10 liquid stocks) from January to June 2025.

4.1 Performance Comparison

Moira significantly outperformed several baselines:

Statistical Baselines: GGR (distance minimization), Correlation, and Cointegration.
RL Baseline: TRIALS (a standard RL framework).
Flat LLM Baseline: A non-hierarchical LLM that attempts to select pairs and execute trades simultaneously.

Key Metrics (Moira vs. Best Baseline):

Annualized Return (AR): Moira achieved 59.11%, compared to 21.38% for the best statistical baseline (Correlation/Cointegration) and negative returns for others.
Sharpe Ratio (SR): Moira achieved 3.791, significantly higher than the statistical baseline (1.218) and the flat LLM (-0.403).
Maximum Drawdown (MDD): Moira maintained a low 1.70%, compared to 5.11% for the statistical baseline and 8.69% for the flat LLM.

4.2 Ablation Studies

The authors conducted ablation studies to validate the necessity of both hierarchical decomposition and prompt adaptation:

Moira-Selection (LLM Selector + Statistical Trader): Replacing the LLM trader with a statistical one resulted in strongly negative returns (-17.10% AR), indicating that semantic pair selection alone is insufficient without adaptive execution.
Moira-Tuning (Statistical Selector + LLM Trader): Using a correlation-based selector with the tuned LLM trader yielded positive returns (17.63% AR) but significantly underperformed the full Moira model. This highlights that static trading behavior cannot fully exploit opportunities identified by semantic selection.
Flat-LLM: The non-hierarchical approach failed to learn, yielding negative returns and high volatility, confirming that joint optimization of pair selection and execution is too complex for a single decision process.

4.3 Learning Dynamics

Analysis of the learning curves showed that performance improved progressively with prompt updates. Early updates established basic discipline (entry/exit rules), while later updates introduced sophisticated risk management (trailing stops, partial profit-taking), leading to smoother equity curves and reduced drawdowns.

5. Significance and Claims

The paper claims that language-driven hierarchical policy optimization offers a principled approach to sequential decision-making problems where:

Abstractions evolve over time.
Feedback is delayed and aggregated.
Explicit credit assignment signals are unavailable.

The authors argue that by using language as a policy representation, the system can achieve targeted adaptation at different temporal scales without the instability associated with gradient-based fine-tuning or the rigidity of traditional HRL. The results suggest that LLMs, when structured hierarchically and optimized via prompt adaptation, can effectively disentangle semantic reasoning from tactical execution, leading to robust financial decision-making.

Limitations Acknowledged:
The authors note that the study is limited to a small asset universe (10 stocks) and a specific market regime. They do not claim robustness under extreme regime shifts or adversarial conditions, and the theoretical properties of textual feedback as an optimization signal remain an open question.