💰 quantitative finance

FactorMiner: A Self-Evolving Agent with Skills and Experience Memory for Financial Alpha Discovery

FactorMiner is a lightweight, self-evolving agent framework that leverages a modular skill architecture and structured experience memory to iteratively discover diverse, high-quality, and interpretable formulaic alpha factors while effectively mitigating redundancy in the expanding search space of quantitative investment.

Original authors: Yanlong Wang, Jian Xu, Hongkang Zhang, Shao-Lun Huang, Danny Dongning Sun, Xiao-Ping Zhang

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanlong Wang, Jian Xu, Hongkang Zhang, Shao-Lun Huang, Danny Dongning Sun, Xiao-Ping Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of quantitative investing, success often hinges on finding a hidden signal within a chaotic sea of market data. Investors look for patterns in price, volume, and trading activity that can predict future movements better than chance. These patterns are often written out as specific mathematical formulas, known as alpha factors, which act like rules for deciding when to buy or sell. The challenge is that the number of possible formulas is so vast that it is impossible for a human to write them all down or test them all manually. Furthermore, as researchers discover more of these rules, they often find that many of them are just variations of the same idea, offering no new insight. This creates a crowded landscape where finding something truly new and useful becomes increasingly difficult. The goal is to build a system that can not only search this vast space efficiently but also learn from its past attempts to avoid repeating mistakes and to discover genuinely unique patterns.

A team of researchers has developed a new approach to this problem called FactorMiner, a self-improving computer agent designed to navigate this complex financial terrain. Instead of relying on a human expert to write every rule or a simple computer program to randomly guess formulas, this system acts like a curious researcher that accumulates experience over time. The core idea is to separate the work of thinking from the work of calculating. The agent uses a large language model to generate ideas for new formulas, but it delegates the actual testing of those formulas to a specialized, high-speed engine. This engine checks if a formula works by looking at real market data, measuring how well it predicts future returns, and ensuring it is not too similar to the thousands of other rules already discovered.

What makes this system unique is its memory. Most automated search methods forget what they learned in previous sessions, often wasting time rediscovering dead ends or generating the same successful patterns over and over again. FactorMiner, however, keeps a structured record of its journey. It stores two types of insights: successful patterns, which are the types of formula structures that have worked well in the past, and forbidden regions, which are combinations of ideas that consistently fail or are too similar to existing rules. When the agent starts a new search, it consults this memory first. It uses the history of what worked and what failed to guide its next guesses, effectively narrowing the search to the most promising areas while avoiding the crowded, unproductive ones.

The researchers tested this system on real-world financial data, including stocks in China and various cryptocurrencies. They found that FactorMiner could build a library of high-quality, diverse rules much more effectively than traditional methods. In their experiments, the system produced a collection of 110 distinct rules that were validated on real market data. These rules were not only accurate in predicting price movements but were also distinct from one another, meaning they captured different aspects of market behavior rather than just repeating the same signal. The system managed to maintain this diversity even as the library grew larger, a feat that many other automated methods struggle to achieve. By continuously refining its search strategy based on what it learned, the agent avoided the "correlation red sea," a state where new discoveries are just copies of old ones, and instead found fresh, interpretable insights.

The process works in a continuous loop. The agent retrieves relevant lessons from its memory, generates a batch of new candidate formulas, and then runs them through a rigorous, multi-stage evaluation. First, it quickly screens out the obvious failures. Then, it checks the survivors against the existing library to ensure they are not too similar to what is already known. If a new rule is better than an existing one but too similar to it, the system allows the new rule to replace the old one, keeping the library up to date. Finally, the most promising rules undergo a full, detailed validation. The results of this entire process are then distilled back into the memory, updating the agent's understanding of what constitutes a good or bad search direction. This cycle allows the system to evolve, becoming more efficient and effective with every round of discovery.

The researchers emphasize that this approach offers a practical solution to the problem of finding new investment signals in a market where easy discoveries have already been made. By combining the creative generation of ideas with a disciplined, memory-guided search, FactorMiner provides a way to scale the discovery of useful financial rules without sacrificing clarity or diversity. The system does not rely on opaque, black-box models that are impossible to understand; instead, it produces clear, human-readable formulas that can be audited and trusted. This work demonstrates that by giving an artificial agent the ability to remember its past experiences and learn from them, it is possible to navigate the immense complexity of financial markets and uncover valuable, interpretable patterns that were previously out of reach.

Technical Summary: FactorMiner

Problem Statement

The paper addresses the critical challenge of formulaic alpha factor mining in quantitative investment. While discovering predictive alpha factors is central to portfolio construction, the task is hindered by three fundamental issues:

Vast Search Complexity: The space of formulaic expressions grows combinatorially with operator compositions and parameters.
Poor Knowledge Accumulation: Traditional search methods (e.g., genetic programming, reinforcement learning) fail to retain insights across exploration sessions, leading to repetitive trials and "knowledge forgetting."
The "Correlation Red Sea": As a factor library grows, the feasible region for new orthogonal factors shrinks rapidly. Existing methods often optimize individual factors in isolation, ignoring how new candidates interact with the global library, resulting in high redundancy and inefficient exploration.

Furthermore, financial practitioners require interpretable, auditable formulas with explicit financial logic, ruling out many black-box neural approaches.

Methodology: FactorMiner Framework

The authors propose FactorMiner, a lightweight, self-evolving agent framework designed to navigate the complex factor space through continuous knowledge accumulation. The system integrates three core components:

1. Modular Skill Architecture

Instead of hard-coding domain logic into the agent's reasoning loop, FactorMiner encapsulates the entire factor mining process as a reusable Agent Skill. This skill includes:

Operator Library: A curated set of 60+ financial operators (e.g., TsRank, Rsqr, VWAP) implemented with GPU-accelerated backends for computational efficiency.
Validation Pipeline: A standardized, multi-stage protocol that decouples generation from evaluation. This prevents "calculation hallucinations" by ensuring all metrics (IC, ICIR) are computed deterministically via code rather than LLM estimation.
Cross-Domain Adaptability: The skill is parameterized to support different markets (e.g., A-shares, Crypto) via configuration files without retraining the agent's reasoning model.

2. Experience Memory

A structured knowledge base that accumulates insights from historical mining sessions to guide future exploration. The memory is formalized through three operators:

Formation: Distills raw mining trajectories into symbolic patterns, categorizing them into Successful Patterns ( $P_{succ}$ $P_{s u cc}$ ) and Forbidden Regions ( $P_{fail}$ $P_{f ai l}$ ).
- Successful Patterns: High-success logical templates (e.g., specific moment regimes or efficiency interactions) that consistently pass quality thresholds.
- Forbidden Regions: Factor families identified as "Red Seas" due to persistent high correlation with existing library members (e.g., simple VWAP deviations).
Evolution: Consolidates redundant entries and discards low-utility information, ensuring the memory remains a compact representation of the search landscape.
Retrieval: During factor generation, the agent retrieves context-dependent memory signals (prompts) to constrain the LLM's policy, effectively shifting the sampling distribution toward orthogonal and promising regions of the program space.

3. The Ralph Loop (Self-Evolving Discovery)

FactorMiner adopts the Ralph Loop paradigm, an iterative refinement cycle:

Retrieve: Fetch relevant patterns from Experience Memory based on the current library state.
Generate: Sample candidate factors using the Agent Skill, guided by retrieved priors.
Evaluate: Execute a Multi-Stage Evaluation Pipeline:
- Stage 1: Fast IC screening on a subset of assets.
- Stage 2: Correlation checks against the existing library to enforce diversity.
- Stage 2.5: Replacement check, allowing high-quality candidates to replace inferior, highly correlated existing factors.
- Stage 3: Batch deduplication.
- Stage 4: Full validation on the entire asset universe.
Distill: Update the Experience Memory with outcomes (successes and failures) to improve future search strategies.

Key Contributions

The paper outlines five primary contributions:

Experience Memory for Agent-Based Mining: Introduces structured memory to enable self-evolution, storing reusable patterns that reduce redundant search compared to memoryless baselines.
Modular Skill Architecture: Designs factor mining as a compositional, reusable skill, separating agent reasoning from deterministic execution to ensure reliability and allow independent optimization.
Lightweight and Efficient System: Builds a high-performance evaluation engine using GPU acceleration, multi-process parallelization, and C-compiled operations, enabling large-scale iterative factor exploration.
Global Factor Library Perspective: Incorporates library admission mechanisms that force the agent to consider how candidates complement the existing library, rather than optimizing in isolation.
Open Factor Library: Provides 110 validated A-share equity factors with explicit formulaic expressions, offering both research insights into market microstructure and practical value for trading.

Experimental Results

The authors evaluated FactorMiner on A-share equities (CSI 300, 500, 1000) and Cryptocurrency markets using 10-minute intraday bars.

Baselines: The system was compared against Alpha101 (Classic and Adapted), Random Formula Exploration, GPLearn, AlphaForge, and AlphaAgent.
Performance: Under a strict protocol where Top-40 factors were selected on CSI 500 (2024) and frozen for evaluation, FactorMiner achieved competitive performance. In the Factor Library setting, it attained an IC of 8.25% and an ICIR of 0.77 on the CSI 500 dataset.
Diversity: Analysis of the resulting 110-factor library showed that most factor pairs were weakly to moderately correlated, with only a few localized clusters of higher dependence, demonstrating the effectiveness of the global library perspective in maintaining orthogonality.
Efficiency: The system successfully handled large-scale iterative evaluation, leveraging GPU acceleration to overcome the computational bottlenecks of standard Python implementations.

Significance and Claims

The paper positions FactorMiner as a practical approach to scalable discovery of interpretable formulaic alpha factors under the constraint of the "Correlation Red Sea."

Shift in Paradigm: It moves beyond treating factor mining as a one-shot search or a purely generative task, framing it instead as a sequential decision problem where the agent learns how to search by accumulating structural knowledge.
Interpretability: By maintaining explicit formulaic expressions, the system addresses the regulatory and risk management needs of financial practitioners who require transparent logic over black-box predictions.
Self-Evolution: The framework demonstrates that an agent can autonomously refine its search strategy over time, avoiding known dead ends and focusing on promising structural patterns without requiring model retraining.

The authors conclude that FactorMiner provides a viable path for constructing diverse, high-quality factor libraries that scale effectively as the search space becomes increasingly constrained by correlation limits.