🤖 AI

From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence

This paper introduces a language-centric framework that unifies diverse multimodal data into an interpretable space of atomic propositions via a global semantic codebook, thereby enabling enhanced reasoning, cross-modal retrieval, and complex composition for applications like autonomous driving.

Original authors: Nadine Chang, Maying Shen, Shizhe Diao, Jialiang Wang, Jingde Chen, Thomas Breuel, Pavlo Molchanov, Rafid Mahmood, Jose M. Alvarez

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Nadine Chang, Maying Shen, Shizhe Diao, Jialiang Wang, Jingde Chen, Thomas Breuel, Pavlo Molchanov, Rafid Mahmood, Jose M. Alvarez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the world. For a long time, scientists have taught these robots by showing them millions of pictures and videos, hoping the robot would "learn" the patterns on its own. This is like giving a student a library of books but never letting them read the words, only asking them to guess the story based on the cover art. The robot builds a giant, invisible map in its brain where similar things are close together, but this map is a mystery even to the humans who built it. We know it works, but we don't know why it works, and if the robot makes a mistake, we can't easily see which part of the map went wrong.

To fix this, researchers are looking for a way to make the robot's thinking more like how humans talk. Instead of just vague feelings or blurry pictures, they want the robot to use clear, simple sentences—like "the car is red" or "the light is green." This paper steps into that conversation, asking: What if we stopped trying to force robots to understand the world through mysterious, invisible maps, and instead gave them a shared dictionary of simple facts? The goal is to turn the messy, confusing world of images and videos into a clean list of statements that anyone (or any machine) can read, check, and combine to understand complex situations.

From Mystery Maps to a Shared Dictionary

Imagine you are describing a chaotic street scene to a friend. You might say, "Wow, look at that big red truck speeding past the wet sidewalk while a dog chases a ball near the stop sign!" That sentence is full of details, but it's also a bit of a mess. If you wanted to find every video of a dog chasing a ball, searching for that whole sentence would be hard because the dog might be chasing a frisbee, or the truck might be blue, or the sidewalk might be dry.

The authors of this paper, working at NVIDIA, propose a new way to organize these descriptions. They call it a Bag of Atomic Propositions (BoAP). Think of "propositions" as the smallest, most basic building blocks of a story—simple, true statements like "dog chasing ball," "truck is red," or "sidewalk is wet."

Instead of letting the robot keep the whole messy sentence in a hidden, confusing part of its brain, this framework breaks every image, video, or text log down into a "bag" of these simple facts. It's like taking a complex LEGO castle and sorting all the bricks into a box, separating the red ones, the blue ones, and the wheels. Once the bricks are sorted, you can easily find every single red brick, or every wheel, no matter what castle it came from.

The Magic Dictionary (The Codebook)

Here is the tricky part: people (and robots) say the same thing in different ways. One person might say "a car is stopped," while another says "the vehicle is waiting." To a computer, these look like totally different things. If you don't fix this, your "bag of facts" gets messy and full of duplicates.

To solve this, the team built a Global Semantic Codebook. Imagine a giant, master dictionary where every possible way of saying "car stopped" is mapped to one single, official entry: "vehicle is waiting."

The process works like this:

Extraction: A super-smart AI (called a Multimodal Large Language Model) looks at a video or image and writes down a list of simple sentences describing what's happening.
Cleaning: The AI strips away the "nuisance" details that don't matter for the big picture, like the specific brand of the car or the exact shade of blue, unless those details are crucial for the task.
Matching: The AI takes every sentence it wrote and checks it against the Master Dictionary. If it finds a match, it swaps the messy sentence for the clean, official version.

Now, every single video in the world, whether it's a self-driving car in Tokyo or a drone flying over a forest in Brazil, is translated into the same language of simple, standard facts.

Why This is a Big Deal

The paper shows that this method does three cool things that the old "mystery map" methods struggle with:

1. It Makes Searching Super Precise
If you want to find a video where a "pedestrian crosses the street" AND "the light is red" AND "the road is wet," the old methods get confused. They might find videos where the light is red but the road is dry, or where a pedestrian crosses but the light is green. Because the new system breaks things down into separate facts, it can mix and match them perfectly. It's like searching a library by checking off specific tags rather than guessing the whole book title. The paper demonstrates this by successfully finding rare, complex scenarios in huge datasets of driving videos and open-world footage.

2. It Reveals What the Robot Doesn't Know
This is perhaps the most powerful part. Because everything is written down in simple facts, you can count them. You can look at your training data (the videos the robot learned from) and see exactly what facts are missing.
For example, the paper analyzed a dataset of driving videos and found that while the robot had seen millions of examples of "cars turning left," it had almost never seen a "car turning left while it is raining and a pedestrian is crossing." The old methods would just say, "Hey, this is a rare car turn," but this new method says, "Hey, you are missing the combination of rain, turning, and pedestrians." This helps scientists know exactly what kind of new data to collect to make the robot safer.

3. It Connects Different Worlds
Because the system translates everything into the same simple language, it can compare a video of a car with a text log of a weather report or a picture of a street sign. They all end up in the same "bag of facts." This allows the robot to understand that a text description of a "wet road" is the same concept as a video showing a wet road, even though one is text and the other is a picture.

The Bottom Line

The paper doesn't claim to have solved all of AI's problems. Instead, it suggests a new way to organize information. It argues that while the old "mystery maps" (dense embeddings) are good at recognizing patterns, they are bad at explaining why something happened or finding specific combinations of events.

By turning the world into a Bag of Atomic Propositions, the authors show that we can make AI more transparent, easier to search, and better at spotting the rare, dangerous situations it might have missed. It's like giving the robot a clear, organized notebook instead of a jumbled sketchbook, allowing us to finally read its mind and understand exactly what it sees.

Technical Summary: From Modalities to Propositions

Problem Statement

Current multimodal intelligence relies heavily on shared embedding spaces (e.g., in Vision-Language Models) to capture semantic information across images, text, audio, and video. While effective for recognition and generation, these high-dimensional representations are largely opaque. Semantic concepts are distributed across dense vectors, making it difficult to identify specific information, interpret model behavior, or perform structured reasoning. Furthermore, real-world understanding is inherently compositional—built from entities, attributes, actions, and relationships—but single dense representations often muddle this richness, obscuring the specific combinations that define complex or rare scenarios.

Methodology

The authors propose a Bag of Atomic Propositions (BoAP) framework, which represents any multimodal observation as a collection of simple, atomic semantic statements. The framework consists of two primary components:

1. Proposition Extraction Pipeline

Given a multimodal observation (e.g., video, image, sensor data), a Multimodal Large Language Model (MLLM) generates a natural language description. This description is processed through three stages to extract Atomic Propositions (APs):

Distillation: The description is reduced to a set of single-clause, declarative sentences, each expressing one fact (e.g., converting "A pedestrian crosses a wet street while a vehicle waits" into separate statements about the pedestrian, the street, and the vehicle).
Proposition Conversion: Each sentence is converted into an AP, defined as the smallest self-contained statement expressing a single fact (e.g., pedestrian crossing street, street is wet). These are structured as triplets of a relation/predicate and noun-anchored arguments.
Nuisance Filtering: Task-irrelevant modifiers (e.g., specific colors, sizes, quantities, or articles) are stripped from the entity arguments to reduce lexical duplication, while preserving the core semantic structure.

2. Global Semantic Codebook Construction

To unify semantically equivalent but lexically different propositions (e.g., "vehicle turning left" vs. "car making a left turn"), the authors construct a global semantic codebook:

Candidate Collection: All filtered APs from the corpus are ranked by frequency, phrase length, and lexicographic order.
Leader-Centric Grouping: High-frequency APs serve as "leaders." Using embedding similarity (via a language embedding model), semantically similar variants are grouped around these leaders.
LLM-Based Verification: An LLM verifies whether the grouped candidates are truly semantically equivalent to the leader, acting as a high-precision filter to ensure only genuinely equivalent concepts are merged.
Canonicalization: The leaders form the Global Semantic Codebook ( $\mathcal{V}$ ), a shared vocabulary of canonical APs. Every observation is mapped to a "Bag of Canonical Propositions" ( $\mathcal{B}(x) \subseteq \mathcal{V}$ ), placing all modalities into a single, interpretable semantic space.

Key Contributions

Language-Centric Representation: A shift from opaque dense embeddings to an interpretable space where observations are expressed as bags of atomic propositions. This unifies diverse modalities (text, video, images) into a single vocabulary.
Explicit Compositionality: The framework enables complex situations to be expressed and reasoned about as explicit combinations of simple concepts. This allows for the identification of rare scenarios defined by the co-occurrence of common concepts (e.g., a specific action + a specific weather condition + a specific infrastructure state).
Measurable Knowledge Coverage: By mapping datasets to distributions over canonical APs, the framework makes semantic content measurable. This allows for the identification of "knowledge gaps" between training and evaluation sets, revealing underrepresented concepts or rare combinations that metadata or caption analysis might miss.

Results and Experiments

The framework was evaluated on three datasets: Physical AI (autonomous driving), NavHard (challenging driving scenarios), and Sekai (open-world exploration).

Semantic Retrieval: The BoAP representation demonstrated superior performance in compositional retrieval compared to holistic caption-based retrieval. As query complexity increased (combining multiple concepts), the similarity scores for AP-based retrieval remained high and accurate, whereas caption-based scores stagnated or diluted. The system successfully retrieved specific, long-tail scenarios (e.g., "pedestrian crossing" + "red light" + "wet road") by matching constituent propositions.
Coverage Analysis: The framework successfully visualized the semantic distribution of ego-vehicle actions, revealing heavy long-tail distributions where common actions dominate and critical edge cases are rare.
Gap Identification: By comparing AP distributions across datasets (e.g., NavTrain vs. NavHard, or Physical AI vs. NavHard), the authors identified specific semantic gaps. For instance, they found that while one dataset covered "yielding and remaining stationary," it missed "continuous progression" scenarios, allowing for targeted data collection strategies to close these gaps.

Significance and Claims

The paper claims that the BoAP framework provides a foundation for interpretable, compositional, and incrementally extensible multimodal intelligence. By moving data into a shared propositional space, the framework enables:

Cross-modal understanding and retrieval without relying on opaque embeddings.
Explicit reasoning over complex situations by composing simple facts.
Systematic data curation by quantifying knowledge coverage and identifying specific conceptual gaps in training data.

The authors emphasize that this approach does not replace dense embeddings but complements them: embeddings retain perceptual robustness, while APs provide the explicit semantic structure necessary for inspection, reasoning, and structured retrieval.