🤖 machine learning

Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting

This paper introduces a multi-scale benchmark dataset and evaluation framework to assess Vision-Language Models' ability to interpret candlestick charts for stock price forecasting, revealing that while these models perform well in persistent trends, they struggle with complex market scenarios and exhibit significant limitations in precise temporal reasoning.

Original authors: Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, Mingwen Liu

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaiqi Hu, Linda Xiao, Shiyue Xu, Ziyi Tang, Mingwen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to predict the stock market. You show it a picture of a stock chart (a "candlestick chart," which looks like a series of colorful bars showing price movements) and ask, "Will this stock go up or down in the next month?"

This paper asks a simple but tricky question: Is the robot actually "reading" the chart, or is it just guessing based on the text you wrote next to it?

Here is the story of their experiment, explained with some everyday analogies.

1. The Problem: The "Cheat Sheet" Confusion

In the past, researchers gave robots a mix of everything: the chart image, a news article, a spreadsheet of numbers, and a summary of the economy.

The Analogy: Imagine taking a math test where you are allowed to use a calculator, a textbook, and a friend who whispers the answers in your ear. If you get an A, did you actually learn math, or did you just listen to your friend?
The Issue: Because the robots had so much text to read, no one knew if they were actually understanding the visual patterns of the stock chart or just reading the text descriptions.

2. The Solution: The "Blindfold" Test

To fix this, the authors built a special test where the robots were blindfolded to text.

The Setup: They gave the robots only the picture of the stock chart. No news, no numbers, no descriptions. Just the visual bars.
The Multi-Scale Twist: They didn't just show one picture. They showed two:
1. A Daily Chart (like looking at a street map to see traffic right now).
2. A Weekly Chart (like looking at a highway map to see the long-term flow of traffic).
The Goal: A good trader looks at both to make a decision. The researchers wanted to see if the robots could do the same thing: combine the "short-term" view with the "long-term" view to predict the future.

3. The Experiment: The Robot Race

They tested several famous AI models (like GPT-4, Claude, and Gemini) against a classic computer program called XGBoost (think of XGBoost as a very disciplined, old-school accountant who only looks at raw numbers).

They asked the robots to predict the stock's return over the next 30 days.

4. The Results: What Did They Find?

🚩 The "Trend Follower" Bias

The robots were great at spotting obvious trends.

The Analogy: If a stock is going up like a rocket, the robot says, "It's going up!" If it's crashing like a stone, the robot says, "It's going down!"
The Catch: They were terrible at the messy middle ground. When the market was sideways or confusing, the robots often got it wrong. They are like a weather app that is perfect at predicting "Sunny" and "Stormy" but terrible at predicting "Cloudy."

⏳ The "Short-Term Memory" Problem

The researchers asked the robots to predict 30 days into the future.

The Surprise: The robots were actually better at predicting what would happen in 5 days.
The Analogy: It's like asking a student to write a thesis on the next 30 years of history. Instead, they just wrote a really good summary of what happened yesterday. The robots are very good at spotting immediate patterns but struggle to plan for the distant future.

🎯 The "Optimist vs. Pessimist" Personalities

Every robot had a different personality:

Claude was very cautious. It rarely predicted a rise unless it was 100% sure (High Precision), but it missed a lot of opportunities (Low Recall). It's like a conservative investor who only buys when it's safe.
GPT-5 Mini was aggressive. It predicted rises often, catching more opportunities but making more mistakes. It's like a gambler who bets on everything.
The Conclusion: There is no "perfect" robot. You have to pick the one that matches your risk tolerance.

5. The Big Takeaway: Why Pictures Matter

The most interesting finding was that the robots (Vision-Language Models) actually did better than the old-school number-crunching program (XGBoost), even though they were only looking at pictures.

The Analogy: Imagine trying to learn a dance.
- The Old Way (XGBoost): You are given a list of numbers describing every step (Step 1: move foot 2 inches left, Step 2: move foot 3 inches right). You have to mathematically figure out the rhythm.
- The New Way (VLMs): You are shown a video of the dance. You can "see" the flow, the shape, and the rhythm instantly.
The Result: The robots could "see" the shape of the market trends in the pictures much better than the computer could calculate them from numbers. The visual format acts like a shortcut for the brain.

Summary

This paper is a reality check for the stock market AI hype.

Yes, AI can "read" charts, but it's not a crystal ball.
It's great at spotting obvious trends (up or down) but bad at predicting the future when things are uncertain.
It has a short attention span, focusing more on the immediate future than the long term.
Visuals are powerful: Showing a robot a picture of a chart helps it understand the market better than just feeding it a spreadsheet of numbers.

The Bottom Line: These robots are excellent tools for a human trader to get a "second opinion" on short-term trends, but they shouldn't be trusted to make long-term investment decisions on their own just yet.

1. Problem Statement

Despite the rapid adoption of Vision-Language Models (VLMs) in financial forecasting, existing research suffers from three critical methodological flaws:

Ambiguous Capability Attribution: Most benchmarks use highly heterogeneous multimodal datasets (combining charts, news text, and tabular data). Consequently, it is unclear whether high performance stems from genuine visual understanding of candlestick patterns or reliance on textual signals.
Lack of Multi-Scale Analysis: Human analysts rely on integrating multiple timeframes (e.g., weekly trends for macro direction and daily charts for entry points). Existing datasets and evaluations typically focus on single-period or tabular inputs, failing to assess a model's ability to synthesize short-term and long-term visual market dynamics.
Insufficient Evaluation Depth: Standard metrics often fail to capture prediction biases, sensitivity to forecast horizons, or the model's behavior across different market regimes (bull, bear, sideways).

2. Methodology

A. Dataset Construction

The authors constructed a novel Multi-Scale Candlestick Chart Dataset to isolate visual reasoning capabilities:

Sources: Historical OHLCV (Open, High, Low, Close, Volume) data from the Chinese HS300 index and US S&P500 index (2015–2025).
Visual Modality: Generated deterministic candlestick images at two scales: Daily and Weekly. Each image includes moving averages (MA5, MA20, MA90) and volume subplots.
Constraints: To prevent data leakage, images are strictly truncated before the prediction date. Each sample contains only visual inputs (no text descriptions of the chart), forcing the VLM to rely solely on pattern recognition.
Target Variable: 30-day forward return ( $r_{30}$ ), calculated as the percentage price change over the next 30 trading days.
Scale: The dataset includes ~193,524 candlestick samples covering diverse market regimes (Bull, Bear, Sideways).

B. Benchmark Tasks & Prompt Engineering

Task Definition: A regression problem where the model predicts a continuous return value $\hat{r} \in [-0.5, 1.0]$ .
Input Structure: Each prompt includes two images (Daily and Weekly charts) for the same stock and date.
Prompt Design: A structured prompt defines the model as a "Stock trend analyst," explicitly instructing it to analyze candlestick bodies/wicks, moving average crossovers, volume spikes, and the consistency between daily and weekly trends.
Output Constraints: Models must output a single numerical score within a specific tag, minimizing hallucination and formatting errors.

C. Evaluation Framework

The study employs a multidimensional evaluation strategy:

Confusion Matrix Diagnostics: Analyzes True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) to derive Accuracy, Precision, Recall, Specificity, and F1 scores. This reveals model "investment philosophies" (e.g., conservative vs. aggressive).
Information Coefficient (IC) Metrics: Measures the linear correlation (and Rank IC) between predicted and actual returns. Key metrics include Mean IC, Median IC, and ICIR (Information Coefficient Ratio) to assess predictive stability and risk-adjusted capability.
Bias Analysis: Quantifies systematic overestimation (optimism) or underestimation (pessimism) of market movements.
Regime-Specific Testing: Evaluates performance specifically in Bull (>70% rising stocks), Bear (>70% falling stocks), and Sideways markets, as well as for stocks with continuous rise/fall patterns.

3. Key Contributions

First Multi-Scale Visual Benchmark: Created the first dataset specifically designed to evaluate VLMs' ability to reason over multi-period candlestick charts without textual confounding.
Isolation of Visual Modality: By removing text and tabular data, the study definitively isolates the marginal value of visual inputs in financial forecasting.
Comprehensive Diagnostic Framework: Introduced a dual-dimensional evaluation combining confusion-matrix-based behavioral analysis with time-series IC metrics to uncover prediction biases and temporal limitations.
Empirical Baselines: Established XGBoost as a rigorous feature-based temporal baseline to compare against VLMs.

4. Experimental Results

A. General Performance

VLMs vs. XGBoost: While XGBoost demonstrated superior stability and average ranking capability, certain VLMs showed potential in specific areas. Claude-sonnet-4-5 (thinking) achieved the best overall performance, showing strong predictive power in both price direction and return ranking. Qwen showed balanced direction prediction, while Gemini2.5-flash offered the most balanced direction prediction.
Market Regime Sensitivity: Most VLMs performed well only under persistent uptrend or downtrend conditions. Performance degraded significantly in common "sideways" market scenarios.
Asymmetric Capability: Models generally demonstrated higher accuracy in identifying bear markets (downside risks) than capturing bull markets (upside opportunities).

B. Prediction Biases & Temporal Reasoning

Systematic Bias: Models exhibited distinct directional preferences (e.g., some were consistently optimistic, others pessimistic). This heterogeneity suggests potential for ensemble methods to cancel out biases.
Time Horizon Failure: A critical finding is that VLMs fail to truly reason over long horizons. Despite being prompted to predict 30-day returns, IC analysis revealed that models correlate significantly stronger with 5-day returns than 30-day returns. This indicates VLMs are capturing short-term technical noise rather than long-term fundamental trends.
Extreme Cases: Models showed varying abilities in identifying continuous rise/fall stocks. For instance, Claude-Haiku excelled at identifying declining stocks, while GPT5mini was better at capturing upward trends.

C. "Peeping" Check

A control experiment comparing GPT4o versions released before and after the test window showed no significant performance difference, providing evidence that models did not "peek" at future data to enhance predictions.

5. Significance and Conclusion

Visual Syntax Advantage: The study posits that VLMs outperform tabular models (like XGBoost) in stability (higher median IC) because candlestick charts externalize domain knowledge into visual syntax. VLMs can transfer priors from natural image recognition (shape, hierarchy) to financial patterns, whereas tabular models must inductively reconstruct these rules from raw numbers.
Practical Implications: VLMs are currently better suited as auxiliary tools for short-term trading signals rather than primary drivers for long-term investment decisions due to their inability to handle cumulative uncertainty over long horizons.
Future Directions: The benchmark provides a foundation for training VLMs specifically on financial visual reasoning, refining model architectures to handle multi-scale temporal dependencies, and developing ensemble strategies that leverage the complementary biases of different models.

In summary, the paper concludes that while VLMs can "read" candlesticks to a degree, their current reasoning is heavily biased toward short-term patterns and specific market regimes, lacking the robust multi-scale temporal integration required for reliable long-term forecasting.