A scalable online machine learning approach for Stock Recommendation

Technical Summary: A Scalable Online Machine Learning Approach for Stock Recommendation

Problem Statement
Stock recommendation systems face a dual challenge: the need to adapt to rapidly changing market conditions (concept drift) while simultaneously maintaining low-latency predictions for end users. Traditional batch-trained models fail to capture real-time shifts in user preferences or market trends, and monolithic architectures often struggle to provide fault tolerance under heavy load. The paper addresses the specific problem of how to build an online stock recommendation system that is both scalable and capable of continuous learning without compromising availability.

Methodology
The proposed solution is a distributed, online deep learning system built on a microservices architecture using Kubernetes, Docker, and RabbitMQ. The core of the system is a hybrid leader-follower architecture:

• Data Ingestion: The system ingests quantitative fundamental and technical data (Earnings Per Share [EPS], MACD-K, MACD-S, current price, and stock name) for S&P 500 stocks via the Alpha Vantage Pro API.
• Model Architecture: The recommendation engine utilizes TensorFlow Recommenders to implement a Multi-Layer Perceptron (MLP). It employs a content-based approach where users provide explicit ratings (1–5) for stocks.
  • The Leader: A single primary model instance continuously trains on streaming data. Upon processing observations, it updates model weights and persists them immediately to Google Cloud Object Storage. In the event of failure, the leader can retrain from logged observations in PostgreSQL.
  • The Followers: Multiple stateless replica models serve user-facing recommendations in parallel. They do not write to disk; instead, they load the latest weights from the leader's storage upon initialization or restart. This design decouples the training loop from the inference loop, enabling horizontal scaling.
• Data Flow: Incoming user requests and stock data are queued via RabbitMQ, which fans out to both leader and followers. The system uses BigQuery for historical stock data storage and PostgreSQL for user portfolio data.
• Training Mechanism: The model updates incrementally. It trains for three epochs on each new observation using the Adagrad optimizer. It utilizes transfer learning and handles concept drift by continuously adapting to new ratings while gradually "forgetting" older data.

Key Contributions
The paper outlines five primary contributions to the field of online financial recommendation systems:

1. Hybrid Leader-Follower Architecture: A novel design that decouples continuous training from low-latency serving. This allows followers to scale horizontally without compromising model freshness and provides a clear failure recovery path via weight persistence and message durability.
2. Explicit Rating-Based Online Learning: Demonstrates that an online deep learning model using explicit user ratings (1–5) can effectively personalize recommendations and capture concept drift in user risk tolerance without requiring full retraining.
3. Minimal Feature Set Efficacy: Shows that meaningful, near real-time recommendations can be generated using only five parameters (EPS, MACD-K, MACD-S, price, and stock name), suggesting that extensive feature engineering is not always necessary for online learning models in this domain.
4. Production-Ready Reference Architecture: Provides a fully implemented, open-source stack (Kubernetes, Docker, RabbitMQ, PostgreSQL, TensorFlow Recommenders) with empirically measured performance characteristics, offering a grounded roadmap for practitioners.
5. Documentation of Limitations: Offers transparency regarding practical failure modes, including leader recovery semantics, cold start behavior, and the absence of service mesh security, guiding future research directions.

Results and Performance
The system was evaluated on its ability to serve recommendations and handle throughput:

• Latency: The system generates stock recommendations in 2–3 seconds per request.
• Throughput: Each follower instance can process up to 300 to 500 portfolio addition requests per second.
• Availability: The architecture ensures high availability; followers can seamlessly recover and reload weights upon failure, and RabbitMQ ensures no data loss during downtime.

Significance and Limitations
The paper positions this work as a practical reference architecture that balances consistency, availability, and scalability in a financial context. It acknowledges that while the system is functional, it operates with specific trade-offs:

• Data Staleness: Due to API rate limits (75 requests/minute on the Pro plan), stock data is refreshed only every 150 minutes, leading to potential staleness in indicators.
• Security: The current implementation lacks a service mesh and Identity and Access Management (IAM) for inter-cluster security.
• Bottlenecks: The leader's write speed to cloud storage (2–5 seconds) and the use of busy-wait locks for thread safety constrain maximum throughput.

The authors conclude that while the system successfully demonstrates the feasibility of scalable online learning for stock recommendations, future iterations must address data latency, security gaps, and retrieval efficiency (e.g., moving from brute-force search to SCANN algorithms) to meet production-grade demands.