💬 NLP

Relational Probing: LM-to-Graph Adaptation for Financial Prediction

This paper introduces "Relational Probing," a method that replaces standard language model heads with a relation head to directly induce relational graphs from hidden states for joint training with downstream stock-trend prediction tasks, demonstrating consistent performance improvements with competitive inference costs using small language models.

Original authors: Yingjie Niu, Changhong Jin, Rian Dolphin, Ruihai Dong

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yingjie Niu, Changhong Jin, Rian Dolphin, Ruihai Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the Stock Market

Imagine you are trying to predict which stocks will go up or down tomorrow. You know that stocks don't move in a vacuum; they are like people at a party. If a big tech company like Nvidia sneezes, its friends (like AMD or Microsoft) might catch a cold, too.

For a long time, financial experts tried to map these connections using static rules (e.g., "These two companies are in the same industry, so they are friends"). But the market changes fast. A static map is like using a paper map from 1990 to navigate today's traffic; it's outdated and full of wrong turns.

Recently, people started using AI (Language Models) to read news and figure out who is talking to whom. But there was a problem: the AI was too chatty. It would write out long paragraphs of text explaining the relationships, which is slow, expensive, and hard to turn into a math problem for a computer to solve quickly.

The Solution: "Relational Probing"

The authors of this paper invented a new method called Relational Probing.

Think of the AI (the Language Model) as a super-smart translator who has read every news article ever written.

The Old Way: You ask the translator, "Who is friends with whom?" and they write you a 5-page essay. Then, a human has to read that essay, highlight the names, and draw lines on a whiteboard. This is slow and prone to human error.
The New Way (Relational Probing): Instead of asking the translator to write an essay, you install a special "Connection Lens" directly onto their eyes. As the translator reads the news, this lens instantly sees the invisible threads connecting the companies and draws a digital map (a graph) directly in their mind.

This map is created while the translator is reading, not after. It's instant, automatic, and perfectly shaped for the next step: predicting the stock price.

How It Works (The Three-Step Dance)

Reading the News (The Input): The AI reads a batch of financial news articles.
The Magic Lens (The Relation Head): Instead of outputting text, a special, lightweight part of the AI (the "Relation Head") looks at the hidden thoughts of the AI and instantly says, "Ah, this article mentions Nvidia and AMD together in a negative context. Let's draw a red line between them." It does this for every company mentioned, creating a daily map of who is influencing whom.
The Prediction (The GAT): A second computer (a Graph Attention Network) looks at this map and the stock prices. It says, "Okay, the map shows Nvidia is connected to AMD, and Nvidia is dropping. Therefore, AMD is likely to drop too." It makes the final prediction.

The Secret Sauce: The best part is that the "Lens" and the "Predictor" are trained together. They learn from each other. If the Predictor makes a mistake, it sends a signal back to the Lens to say, "Hey, you drew the wrong line there; try again." This makes the whole system smarter over time.

Why "Small" Models?

The paper focuses on Small Language Models (SLMs).

The Analogy: Think of a massive AI (like a giant supercomputer) as a Olympic swimmer. They are incredibly strong but require a massive pool, a huge team of coaches, and millions of dollars to train.
The SLM: The authors use "Small" models (like the Qwen3 series). Think of these as fit local swimmers. They aren't as powerful as the Olympians, but they are fast, cheap, and can be trained in a standard backyard pool (a single computer graphics card).
The Result: The paper shows that these "local swimmers" are actually perfect for this job. They are fast enough to process news in real-time and smart enough to draw the connection maps accurately, without needing a supercomputer.

What Did They Find?

Better Maps: The maps drawn by their "Connection Lens" were much better than maps made by just counting how often two stocks appeared in the same sentence (the old "co-occurrence" method). The Lens understood context (e.g., knowing that two companies mentioned together might be rivals, not friends).
Size Matters (But Only a Little): Bigger models (4 Billion parameters) did slightly better than smaller ones (0.6 Billion), but the difference wasn't huge. This is great news because it means you don't need the most expensive AI to get good results.
Speed and Cost: By skipping the "write an essay" step and going straight to "draw a map," the system is much faster and cheaper to run.

The Takeaway

This paper is about making AI less chatty and more practical for finance. Instead of having an AI write a story about the market, they taught it to instantly draw a relationship map that helps predict stock trends. It's like upgrading from a slow, manual cartographer to a GPS that updates the map in real-time as you drive.

In short: They built a system that reads financial news, instantly figures out how companies are connected, and uses that knowledge to guess where the stock market is going—all without needing a supercomputer or waiting for the AI to finish writing a novel.

1. Problem Statement

Financial market prediction relies heavily on understanding relationships between stocks (e.g., sector co-movement, supply chain dependencies). While recent approaches use Large Language Models (LLMs) to extract these relationships from news text, existing methods suffer from significant limitations:

Autoregressive Overhead: Prompt-based pipelines require autoregressive decoding to generate structured text (e.g., JSON), which is computationally expensive and introduces latency.
Decoupled Optimization: Graph construction is typically a separate pre-processing stage, decoupled from the downstream prediction task. This prevents the graph structure from being optimized specifically for the prediction goal.
Fragility: Instruction-following LLMs often fail to adhere to strict formatting constraints required for graph construction, leading to parsing errors.
Static/Noisy Baselines: Traditional methods rely on static pre-defined graphs (e.g., industry sectors) or simple co-occurrence, which fail to capture dynamic, context-dependent semantic relationships in real-time news.

2. Methodology: Relational Probing

The authors propose Relational Probing, an end-to-end framework that replaces the standard language modeling head of an LLM with a lightweight Relation Head (RH). This allows the model to induce a relational graph directly from hidden states, trained jointly with a downstream Graph Attention Network (GAT).

Core Components:

News Encoding:
- News articles are tokenized and encoded by an upstream Small Language Model (SLM) backbone (e.g., Qwen3) to produce contextual token representations ( $H$ ).
Relation Induction (The Relation Head):
- Instead of generating text, the RH maps hidden states to an adjacency matrix.
- Mechanism: Learnable ticker embeddings ( $E$ ) attend to token states via scaled dot-product attention to create news-conditioned ticker representations ( $Z$ ).
- Interaction Scoring: Pairwise interactions between tickers are computed using a bilinear map on the reduced space ( $Z W_{bi} Z^T$ ), followed by layer normalization.
- Graph Construction: Daily edge attributes are aggregated from all news articles, thresholded to create a sparse adjacency matrix ( $A$ ), and combined with edge weights to form a daily weighted graph ( $G_t$ ).
Downstream Prediction:
- A Graph Attention Network (GAT) takes the induced graph and historical trading features (encoded via LSTM) as input.
- The GAT updates node representations based on the induced edges and predicts stock trends (Up, Down, Neutral).
Joint Training:
- The entire pipeline (SLM $\to$ RH $\to$ GAT) is trained jointly. Gradients from the downstream prediction loss backpropagate through the GAT and RH into the SLM, aligning the induced graph structure directly with the financial prediction objective.

Operational Definition of SLMs:

To ensure reproducibility and portability, the authors define Small Language Models (SLMs) as models capable of being fine-tuned end-to-end (including the RH and downstream task) on a single 24GB GPU under fixed batch-size and sequence-length budgets.

3. Key Contributions

Relational Probing: Generalizes linear probing to a "Relation Head" that converts hidden states directly into relational graphs, eliminating the need for constrained decoding and parsing.
Joint Optimization: Demonstrates that training the graph induction mechanism jointly with the downstream predictor yields better task-aligned graph structures than two-stage pipelines.
SLM Benchmarking: Provides a standardized operational definition for SLMs and benchmarks the Qwen3 family (0.6B, 1.7B, 4B) within a controlled single-GPU setting.
Efficiency: Removes the latency and cost associated with autoregressive generation while maintaining high-quality semantic extraction.

4. Experimental Results

Experiments were conducted on a financial news dataset with stock trend prediction as the downstream task, comparing Relational Probing against a Co-occurrence Baseline (graphs built on simple word co-occurrence).

Performance Metrics: Evaluated using Accuracy, Macro F1, Matthews Correlation Coefficient (MCC), and AUC.
Key Findings:
- Superiority over Baselines: Relational probing consistently outperformed the co-occurrence baseline across all metrics.
  - Example (Qwen3-0.6B): Macro F1 improved from 0.2831 (Co-occurrence) to 0.3171 (+0.0340); MCC improved from 0.0143 to 0.0312.
- Model Scaling: Performance scaled with SLM capacity (0.6B $\to$ 1.7B $\to$ 4B), though with diminishing returns. The 4B model achieved the best results (Accuracy: 0.5705, Macro F1: 0.3272).
- Ablation Study:
  - Full Attention vs. Limited: The proposed "Full" attention RH (considering all token-ticker interactions) outperformed "Limited" (restricted to mentioned tickers) and "Pooling" strategies, proving the necessity of semantic fusion.
  - Input vs. Generated Tokens: Using only input sequence hidden states (Input-only) performed better or equal to using generated tokens (Input+Gen), while being significantly faster computationally.
Addressing Class Imbalance: The study noted that while absolute metric gains were modest (typical for noisy financial data), the improvements were significant in identifying minority classes (Up/Down trends) rather than just the majority class (Neutral).

5. Significance

Paradigm Shift: Moves financial NLP from "Text Generation $\to$ Parsing $\to$ Graph" to "Hidden States $\to$ Direct Graph Induction," significantly reducing pipeline complexity and inference cost.
Task Alignment: By jointly training the graph construction with the prediction task, the model learns to extract predictive relationships rather than just linguistic ones.
Practicality: The framework is designed for resource-constrained environments (single GPU), making advanced graph-based financial prediction accessible without requiring massive LLM clusters.
Reproducibility: The operational definition of SLMs provides a clear standard for future research in efficient, end-to-end financial modeling.

In conclusion, Relational Probing offers a robust, efficient, and end-to-end solution for integrating unstructured financial text into structured graph models, outperforming traditional baselines while maintaining computational feasibility.