Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)
This Fall 2023 study from George Washington University demonstrates that a pipeline using Falcon-7B with Summarize Chains effectively automates financial news summarization for ten major companies, outperforming both RAG-based approaches and a Lead-3 baseline while highlighting persistent challenges like hallucination in smaller models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every day, the financial world generates a flood of information. Thousands of articles appear online, each detailing the movements of companies, the decisions of executives, and the shifting tides of the stock market. For an investor or an analyst trying to keep track of even a small handful of these companies, reading every single story is an impossible task. Missing a crucial detail can mean the difference between a smart investment and a costly mistake. This is where the field of natural language processing steps in, specifically the branch known as text summarization. For decades, computers have been taught to read long documents and extract the most important sentences. More recently, a new generation of artificial intelligence, called large language models, has emerged. These are systems trained on vast amounts of human writing that can not only pick out key sentences but also rewrite information in their own words, creating a fresh, concise narrative from a mountain of text. The question facing researchers became whether these powerful tools could be trusted to do this work for the high-stakes world of finance, where accuracy is non-negotiable and the data comes in two very different forms: the flowing stories of news articles and the rigid, numerical tables of stock prices.
In the autumn of 2023, a researcher at George Washington University set out to test this idea. The goal was to build a system that could automatically gather news, company backgrounds, and stock market data for ten major corporations and then produce a single, easy-to-read daily briefing. The challenge was twofold. First, the system had to handle the sheer volume of text. Second, and perhaps more difficult, it had to make sense of numbers. Large language models are excellent at understanding language, but they often stumble when presented with raw spreadsheets of stock prices and trading volumes. They might misread a column, calculate a percentage incorrectly, or simply ignore the numbers entirely. To solve this, the researcher developed a simple but clever bridge. Before feeding the data to the artificial intelligence, a computer program converted the raw numbers into plain English sentences. Instead of showing the model a table with a date and a price, the system wrote a sentence like, "On October 4th, the stock closed at 261.16, up 5.93 percent with a 27.20 percent increase in trading volume." This allowed the language model to focus on summarizing the story without having to do math, which is a task it is not naturally good at.
The researcher then tested two different ways of organizing this information for the computer to read. The first method, called the Summarize Chains approach, works like a relay race. The system breaks the long collection of news and background information into smaller chunks. It asks the computer to summarize the first chunk, then the second, and so on. Finally, it takes all those small summaries and combines them into one master summary. The second method, known as retrieval-augmented generation, is more like a librarian. When the system needs to write a summary, it searches through the entire collection of documents to find the specific sentences that are most relevant to the current question, and then it gives only those selected sentences to the computer to work with. The study evaluated three different open-source artificial intelligence models using these methods, alongside a more expensive, proprietary model for the stock data summaries.
The results offered a clear picture of what works and what does not. The most successful approach turned out to be the Summarize Chains method using a specific open-source model called Falcon-7B. This combination managed to cover every major news event accurately, preserving important details like the exact number of employees laid off by a tech giant, and it did so without repeating itself or making things up. In contrast, the librarian-style retrieval method caused significant problems. When the system tried to retrieve too many relevant sentences at once, the computer became confused, repeating the same three news items dozens of times in a single summary. In other cases, the retrieval method led the computer to mix up facts, such as attributing a fine paid by one company to a completely different one. These errors, known as hallucinations, happened more often when the computer was given too much information to sort through at once.
The study also confirmed that the strategy of converting numbers into sentences worked perfectly. When the computer was given the pre-written sentences about stock prices, it produced summaries that were factually accurate across all six companies tested, with no invented numbers or incorrect calculations. This proved that moving the math out of the artificial intelligence's hands and into a simple computer script was the right move. While the retrieval method showed promise in theory, the study found that in practice, it introduced too many errors and repetitions to be reliable for financial reporting without significant changes. The researcher built a working dashboard that allows users to see stock charts and read these automated summaries, demonstrating that the system is practical enough to be used as a real tool. The work suggests that while artificial intelligence can indeed automate the reading of financial news, the way the information is fed to the machine matters just as much as the machine itself. The most reliable path forward, for now, is to let the computer do what it does best—understand language—while letting a simple script handle the numbers and organize the information in a way that prevents the machine from getting overwhelmed.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.