FactorEngine: A Program-level Knowledge-Infused Factor Mining Framework for Quantitative Investment

FactorEngine is a program-level, knowledge-infused framework that leverages LLM-guided search and a multi-agent extraction pipeline to discover executable, auditable alpha factors with superior predictive stability and portfolio performance compared to existing symbolic and neural methods.

The Gist

Imagine you are trying to find the perfect recipe for a dish that will make you rich, but the ingredients (the stock market) change flavor every single day. Sometimes it's spicy, sometimes bland, and sometimes it's toxic.

For decades, investors have tried to find these "magic recipes" (called Alpha Factors) using two main methods:

1. The Manual Chefs: Humans write strict rules (like "if the price drops, buy"). This is slow and hard to scale.
2. The Black Box AI: Neural networks guess the recipe by tasting millions of dishes. They are good at guessing, but you have no idea why they chose that recipe, and they often fail when the ingredients change.

FactorEngine (FE) is a new, third way. Think of it as a Super-Chef Robot that doesn't just guess or follow strict rules. Instead, it writes its own cooking code, learns from its mistakes, and constantly improves its recipes.

Here is how FactorEngine works, broken down into simple analogies:

1. The "Bootstrapping" Phase: Reading the Cookbook

Before the robot starts cooking, it needs inspiration.

• The Old Way: Humans read financial reports and try to guess what the robot should do.
• The FactorEngine Way: The robot has a team of AI Agents that read thousands of financial reports. They don't just summarize them; they extract the core idea (e.g., "When volume spikes and price jumps, the stock is likely to keep going up") and immediately turn that idea into executable computer code.
• The Analogy: Imagine a team of translators who read a chef's handwritten notes and instantly turn them into a working robot program. This gives the robot a massive head start with "knowledge-infused" recipes.

2. The "Evolution" Phase: The Macro-Micro Dance

This is the heart of the system. FactorEngine splits the work into two distinct teams to avoid getting stuck or wasting time.

• The Macro Team (The Creative Architect):

  • Who: Large Language Models (LLMs), like the smartest chefs in the world.
  • Job: They look at the big picture. They ask, "What if we change the logic? What if we look at the stock's behavior over 3 days instead of 1?" They use a "Chain of Experience" (a memory of past successes and failures) to guide their ideas.
  • Analogy: This is the architect redesigning the kitchen layout. They decide what to cook and how to approach the problem.

• The Micro Team (The Precision Tuner):

  • Who: A fast, automated math engine (Bayesian Optimization).
  • Job: Once the Architect proposes a new recipe, the Tuner fine-tunes the numbers. "Should the window be 3 days or 3.5 days? Should the decay factor be 0.9 or 0.95?"
  • Analogy: This is the sous-chef adjusting the salt and pepper. They don't change the recipe; they just make sure the measurements are perfect.

Why separate them?
If you ask the Creative Architect to also count the salt, they get distracted and slow down. By separating the "Big Ideas" (LLM) from the "Fine Tuning" (Math), FactorEngine runs much faster and smarter.

3. The "Island" Strategy: Avoiding Groupthink

In nature, if a species lives on one island, they might all evolve the same way and die out if the environment changes.

• FactorEngine's Solution: It runs multiple independent "Islands" of evolution at the same time.
• The Migration: Every few rounds, the best recipes from one island are sent to the others.
• The Analogy: Imagine five different cooking competitions happening simultaneously. Every week, the winner of Competition A sends their best dish to Competition B. This ensures that a brilliant idea discovered in one group spreads to everyone, preventing the system from getting stuck in a "local optimum" (a good but not great solution).

4. The "Verification" Loop: Learning from Failure

Most AI systems only learn from what works. FactorEngine is special because it learns from failures.

• If a recipe fails (the code crashes or loses money), the system records why it failed.
• The "Chain of Experience" remembers these failures so the Architect doesn't make the same mistake twice.
• Analogy: It's like a student who keeps a diary of every wrong answer on a test, not just the right ones, so they never miss that specific question again.

The Results: Why It Matters

When tested against the best existing methods (like human-made factors or other AI agents):

• Better Predictions: It found signals that were more accurate (higher IC).
• More Profit: It generated significantly higher returns (up to 126% more than some baselines).
• More Stable: It didn't crash as hard when the market turned (lower drawdown).
• Diverse: It found a wider variety of unique strategies, rather than just copying the same idea over and over.

Summary

FactorEngine is like a self-improving, multi-agent factory for investment strategies.

1. It reads human knowledge and turns it into code.
2. It separates "Big Creative Ideas" from "Mathematical Tweaking" to be super efficient.
3. It runs parallel experiments to ensure diversity.
4. It learns from its mistakes to build better, more robust strategies.

Instead of just guessing the market or following rigid rules, FactorEngine writes its own adaptable code, making it a powerful tool for navigating the chaotic, noisy world of finance.

Technical Summary

1. Problem Statement

The paper addresses the challenge of Alpha Factor Mining: the automated discovery of predictive signals from noisy, non-stationary financial market data. Existing approaches suffer from three critical limitations:

• Bounded Expressiveness: Traditional symbolic methods (e.g., Genetic Programming) rely on predefined operator sets, limiting the search space and resulting in fragile factors that struggle with regime shifts.
• Limited Diversity and Stability: Current methods lack mechanisms to effectively integrate unstructured financial knowledge (e.g., research reports) into executable code, leading to a lack of diverse, theoretically grounded factors.
• Inefficient Evolution Pipelines: There is a significant speed mismatch between Large Language Model (LLM) generation (logic discovery) and evaluation (backtesting). Existing frameworks often force LLMs to handle both logic and parameter optimization, creating computational bottlenecks and high costs.

The goal is to create a framework that produces executable, auditable, and interpretable factors that are computationally tractable at scale and robust against market changes.

2. Methodology: FactorEngine (FE)

FactorEngine is a program-level factor discovery framework that treats alpha factors as Turing-complete Python code rather than fixed symbolic expressions. It achieves efficiency and effectiveness through three key separations and a closed-loop pipeline.

Core Architecture

The system consists of three decoupled modules:

1. Bootstrapping Module (Knowledge Infusion):

   • Transforms unstructured financial reports and expert knowledge into executable factor programs.
   • Uses a closed-loop multi-agent pipeline: Extraction (distilling core ideas from reports), Verification (checking logical consistency), and Code Generation (converting pseudocode to executable Python).
   • This creates a "knowledge-infused" initial pool, seeding the evolution process with high-quality, domain-grounded hypotheses.

2. Evolution Module (Macro-Micro Co-Evolution):

   • Macro-Level (Logic Evolution): LLM agents perform "macro mutations" (structural changes, new logic) guided by a Chain of Experience (CoE). The CoE captures historical trajectories, including failures and transient fluctuations, allowing the LLM to learn from past mistakes and steer exploration toward robust directions.
   • Micro-Level (Parameter Optimization): Instead of asking the LLM to tune parameters, FE uses Bayesian Optimization (e.g., TPE, Gaussian Processes) on local computational resources. This separates semantic reasoning from numerical tuning.
   • Resource Separation: LLMs are used only for code generation and logic refinement (Phase 1), while heavy backtesting and parameter search (Phase 2) run on local parallel workers without LLM involvement.
   • Multi-Island Strategy: To combat the stochastic nature of LLMs, FE runs multiple independent evolution "islands" that periodically migrate top-performing programs, ensuring diversity and preventing premature convergence.

3. Integration Module:

   • Selects elite factors from the evolved pool based on a weighted fitness score (combining IC, ICIR, Rank IC, and Rank ICIR).
   • Aggregates these factors into a multi-factor model for backtesting and portfolio construction.

3. Key Contributions

• Program-Level Hyper-Heuristic Framework: Unlike symbolic search, FE evolves Turing-complete programs, allowing for complex control flows, conditional logic, and iterative computation, significantly expanding the search space.
• Macro-Micro Co-Evolution: The framework decouples logic discovery (LLM-driven) from parameter tuning (Bayesian search), solving the efficiency bottleneck and local optima issues associated with joint optimization.
• Knowledge-Infused Bootstrapping: A novel pipeline that converts unstructured financial reports into executable code, enabling the system to leverage prior economic knowledge and transparent rationales.
• Experience-Guided Refinement: The introduction of a "Chain of Experience" allows the system to learn from failure trajectories, not just successful nodes, mimicking human-like reasoning to recover from performance dips.

4. Experimental Results

The authors evaluated FE on real-world OHLCV data (CSI300 and CSI500 markets) against baselines including GPlearn, Neural Networks (LSTM, Transformer), and agent-based methods (AlphaAgent, RD-Agent).

• Predictive Performance: FE significantly outperformed all baselines.
  • IC (Information Coefficient): FE-report achieved an IC of 0.0474 (CSI300), a 58% improvement over Alpha158.
  • Excess Annual Return: FE-report achieved 18.99% (CSI300), a 126% increase over Alpha158.
  • Stability: FE factors showed superior stability in Rank IC and Rank ICIR across different time horizons and lag lengths.
• Portfolio Performance:
  • FE achieved higher Sharpe Ratios and Information Ratios with lower Maximum Drawdown (MDD) compared to baselines.
  • In the CSI300 market, FE-report reduced MDD to 12.61% while maintaining high returns.
• Factor Diversity:
  • Using Multidimensional Scaling (MDS), FE-generated factors showed a "circular dispersion" pattern, indicating low redundancy and high independence compared to the clustered factors of other agent-based methods.
  • FE retained a 57.1% keep ratio of effective factors, significantly higher than AlphaAgent (21.9%) and RD-Agent (3.9%).
• Efficiency:
  • FE reduced runtime from ~48 hours (RD-Agent) to 0.5 hours for 200 iterations.
  • It achieved a 99% executable code ratio with only 32% debug API calls, demonstrating high operational efficiency.

5. Significance

FactorEngine represents a paradigm shift in quantitative investment research by bridging the gap between domain knowledge (financial reports) and automated code generation.

• Scalability: By separating logic generation from heavy computation, it makes large-scale factor mining computationally feasible.
• Interpretability & Auditability: As factors are generated as executable Python code with clear economic rationales derived from reports, they are easier to audit and understand than "black-box" neural networks.
• Generalizability: The framework is a gradient-free optimization method for discrete, structured search spaces, making it potentially applicable to other black-box optimization problems beyond finance where evaluation is expensive.

In summary, FactorEngine demonstrates that combining LLMs for semantic logic discovery with Bayesian methods for parameter tuning, all grounded in financial knowledge, yields factors that are not only more profitable but also more robust and diverse than current state-of-the-art methods.