Don't Just Listen, Try Planning: Graph-based Retrieval-Generation Agent for Long-form Audio Meeting Understanding

Imagine sitting in a long, crowded meeting where voices overlap, people interrupt one another, and the conversation drifts from a budget discussion to a sudden outburst of frustration. If you were asked later to explain exactly why a specific person raised their voice at the nineteen-minute mark, or to trace how a decision made at the start of the hour influenced a complaint made twenty minutes later, you would rely on more than just the words spoken. You would remember the tone, the volume, the pauses, and the way the conversation flowed from one person to another. For decades, computers have struggled to do the same. While machines have become excellent at turning speech into text, they often fail to understand the full context of long, complex conversations. They tend to treat a meeting as a flat list of sentences, losing the crucial details of who said what, when they said it, and how they said it. This limitation becomes a major hurdle when trying to answer questions about hours-long recordings, where the answer to a simple question might be hidden deep within a web of interactions that span the entire duration.

Researchers at Soochow University in China have developed a new approach to solve this problem, creating a system that does not just listen to audio but actively plans how to find the answer. They began by acknowledging that existing artificial intelligence models, which are designed to process speech, often discard valuable acoustic information like voice volume and emotion in favor of the raw text. Furthermore, these models struggle to remember details from the beginning of a long recording when they are trying to answer a question about the end. To address this, the team first built a new dataset called LongAudioQA, which contains hundreds of questions about real-world meetings, ranging from simple factual queries to complex questions that require understanding the emotional state of a speaker or the logical connection between two distant parts of a conversation.

The core of their solution is a system they call a graph-based retrieval-generation agent. Instead of feeding the entire audio file into a single model at once, the researchers first break the meeting down into a structured map. In this map, every sentence spoken becomes a node, and the connections between them are drawn based on who is speaking, when they are speaking, and what they are talking about. This structure allows the computer to see the meeting not as a stream of words, but as a network of relationships. When a user asks a question, the system does not simply search for matching words. Instead, it acts like a human expert who plans their search. It breaks the question down, decides which parts of the map to explore, and then moves step-by-step through the connections to gather evidence.

If the initial search does not provide enough information, the system has the ability to reflect on its own progress. It can realize that it is missing a piece of the puzzle, such as the specific tone of a voice or the context of a previous statement, and then re-plan its search to look for that missing piece. This cycle of searching, gathering, and reflecting continues until the system feels confident it has found the correct answer. This method allows the system to handle questions that require connecting dots across time, such as identifying why a speaker sounded angry at a specific moment by linking that moment to a previous promise that was broken.

The results of their testing show that this planning-based approach significantly outperforms current state-of-the-art models that rely on simple text search or direct audio processing. In tests involving meetings lasting over thirty minutes, the new system was able to answer complex questions with much higher accuracy, particularly when the questions required understanding the emotional tone of the speakers or tracking the flow of a conversation over a long period. The researchers found that by explicitly modeling the relationships between speakers and time, and by allowing the system to plan its own search strategy, they could overcome the common problem of "context fragmentation," where important details are lost because they are too far apart in the recording.

This work suggests that for computers to truly understand long-form human conversation, they need more than just powerful processors; they need a way to organize information that mirrors how humans naturally navigate complex social interactions. By treating a meeting as a structured map and giving the computer the ability to think through its search process, the researchers have created a system that can listen to a meeting, understand the relationships within it, and provide answers that are grounded in the actual evidence of the recording. While the system still faces challenges with extremely noisy recordings, its ability to plan and reflect marks a significant step forward in making artificial intelligence capable of understanding the full depth of human dialogue.