🤖 AI

NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

The paper introduces NoRA, a visual first-person benchmark comprising 1,420 annotated video clips that evaluates multimodal systems on their ability to generate and justify reasonable actions through explicit fact-reason-action support graphs, revealing that current models struggle to construct the full action space and correctly bind actions to visible evidence despite often identifying plausible individual actions.

Original authors: Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar

Published 2026-06-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Knowing vs. Doing

Imagine you are teaching a robot to walk through a busy park.

Old Way: You show the robot a list of three options: "A) Walk left," "B) Walk right," or "C) Sit down." The robot picks "B." You check if "B" is the "correct" answer.
The Paper's Critique: This is like a multiple-choice test. In real life, nobody hands you a menu of three options. You have to look at the situation, figure out what's happening, come up with your own list of possible moves, and then pick the best one.

The authors argue that current AI models are good at picking the right answer from a menu (like a student guessing on a test), but they are bad at figuring out the menu itself and explaining why they picked a specific dish.

The Solution: NORA (The "Reasoning Gym")

The authors created a new test called NORA (Normative Reasoning in Action). Instead of asking an AI, "What should I do?" and checking if the answer matches a pre-written key, NORA asks the AI to do three things simultaneously:

See the scene: Look at a video clip from a first-person perspective (like a GoPro on someone's head).
Build a menu: Generate a list of reasonable things to do next.
Justify the choice: For every action on the list, explain why it makes sense based on what is actually visible in the video.

The Analogy: Think of NORA not as a multiple-choice quiz, but as a cooking competition.

Old Tests: The judge gives you three ingredients and asks, "Which one is the best?"
NORA: The judge shows you a kitchen with a messy counter and a hungry guest. You have to:
- Identify the ingredients on the counter (Facts).
- Come up with a few different meals you could cook (Candidate Actions).
- Explain why "Pasta" is a good choice because "The guest is hungry" and "We have tomatoes" (Reasons).
- Crucially: If you say "I'll make a steak," but there is no meat on the counter, you fail, even if "steak" is a delicious idea.

How They Measure Success: The "Support Graph"

The paper introduces a specific way to grade the AI called a Grounded Reasonableness Score. They don't just look at the final answer; they look at the "support graph."

Imagine a tree:

The Roots (Facts): What is actually visible? (e.g., "There is a child running," "The ground is wet").
The Trunk (Reasons): Why does this matter? (e.g., "The child might slip," "It's dangerous to run on wet grass").
The Fruit (Actions): What should we do? (e.g., "Stop the child," "Move to dry grass").

The Score:

Action Alignment: Did the AI come up with a good list of fruits?
Factual Grounding: Are the roots real? (Did the AI hallucinate a child that isn't there?)
Support Binding: Is the fruit connected to the right roots? (Did the AI say "Stop the child" because of the wet grass, or because of a dog that isn't there?)

What They Found: The "Plausible but Wrong" Trap

The researchers tested 12 different AI models (including big names like GPT-5 and Gemini) using this new gym. Here is what they discovered:

They are good at describing the room: Most AIs can correctly identify visible facts (e.g., "There is a lawn mower").
They are okay at picking a fruit: They often pick a plausible action (e.g., "Keep mowing").
They fail at the "Menu" and the "Why":
- Missing the Menu: They struggle to generate a full list of reasonable options. They often miss alternative actions that a human would consider.
- Loose Connections: They often pick a good action but attach the wrong reason to it. For example, they might say "Keep mowing" because "It's a nice day," when the real reason (visible in the video) is "The grass is long and needs cutting."

The Metaphor: Imagine a student who raises their hand and says, "The answer is 42!"

Old Test: If 42 is the right answer, they get an A.
NORA Test: The teacher asks, "Show your work." The student says, "Because 6 times 7 is 42."
- If the problem was actually "What is the square root of 1764?", the student gets an A for the answer but fails the test because they didn't show the right reasoning for the specific problem.
- NORA found that current AIs are great at shouting "42!" but terrible at showing the math that proves why 42 is the right move for this specific video.

The Three "Prompt" Styles

To test how the AI thinks, the researchers asked the same questions in three different ways (like changing the rules of the cooking competition):

Direct: "Just tell me what to do." (No explanation allowed).
Deliberate: "Think out loud. List a few options and explain them, then pick one."
Structured: "Fill out this specific form: List Facts, List Reasons, List Actions, then Pick One."

The Result: The "Structured" approach (filling out the form) helped the best AI models perform better. It forced them to slow down and connect their facts to their actions. However, for some other models, forcing them to fill out a form actually made them worse, as if the rigid rules confused them.

The Bottom Line

The paper concludes that while AI is getting better at "seeing" and "picking," it still struggles with justified reasoning. It can often guess the right move, but it can't reliably explain why that move is right based only on what it sees in the video.

NORA is a tool to measure this gap. It shifts the question from "Can the AI pick the right answer?" to "Can the AI build a logical, visible, and defensible argument for its action?" Currently, the answer is: "They are getting there, but they are still missing the full picture."

Technical Summary: NORA – Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

Problem Statement

Current evaluations of Large Language Models (LLMs) and Vision-Language Models (VLMs) regarding normative competence—specifically the ability to navigate social norms and practical reasons for action in real-world environments—are insufficient. Existing benchmarks typically rely on two flawed paradigms:

Text-only moral judgment: Assessing models on explicitly described textual cases, which fails to test the ability to isolate relevant features from visual evidence.
Multiple-Choice Questions (MCQs): Even in visual settings (e.g., EGONORMIA), models are presented with a fixed menu of options. This approach tests answer-matching rather than the capacity to construct a reasonable action space from scratch.

The paper argues that these methods remain on the "judgment side" of the knowing–doing gap. In practice, agents must identify normatively relevant features from first-person video, reason about them, and generate a justified action without a pre-defined list of candidates. Current models often fail to construct the full space of reasonable alternatives or bind selected actions to the correct local support (facts and reasons).

Methodology: The NORA Benchmark

The authors introduce NORA (Normative Reasoning in Action), a benchmark designed to evaluate grounded reasonableness. This concept is defined as a model's ability to identify salient scene facts, represent reasons for or against candidate actions, and select an action supported by those reasons within a socially situated context.

Dataset Construction

NORA is built upon clips from the Ego4D dataset, annotated under philosopher-reviewed instructions. The dataset comprises 1,420 annotated video clips divided into two tiers:

HUMANGOLD-190: A 190-clip set verified by human annotators.
LLMSILVER-1230: A 1,230-clip set validated by LLMs (GPT-5.4 and Gemini-3.1-Pro) against the human gold standard to ensure scale.

Annotation Schema:
Each clip is annotated as a structured decision problem following three principles:

First-person perspective: Grounded strictly in visual evidence available in the scene.
Fact–Reason–Action order: Annotations record visible facts, fact-based reasons, and reason-based candidate actions in a strict sequence, avoiding post-hoc justification.
Reasonable-action pluralism: Multiple actions may be reasonable if grounded in visible evidence and supported by reasons not defeated by stronger contextual considerations.

Annotations are represented as action-rooted support graphs ( $G = \{G_a : a \in A\}$ ), where edges encode local support relations ( $f \to r \to a$ ).

Evaluation Protocol

The evaluation measures grounded reasonableness by comparing model outputs against the reference support graphs. The metric decomposes into three components:

Action Alignment ( $Q_A$ ): Semantic recovery of candidate next actions.
Factual Grounding ( $Q_F$ ): Recovery of visible scene facts.
Support Binding ( $Q_S$ ): Whether facts and reasons are correctly bound to the appropriate actions.

The primary aggregate metric is the geometric mean of these three components: $R = (Q_A \cdot Q_F \cdot Q_S)^{1/3}$ . This ensures a model cannot achieve a high score by merely generating a plausible action if it fails to ground it in facts or bind it to correct reasons.

Prompting Regimes:
The study evaluates 12 VLMs under three distinct prompting conditions to test different levels of justification:

Direct (Simple): Asks for a next action with minimal explanation (simulating deployment).
Deliberate (Medium): Requires free-form analysis of candidate actions and reasons before selection.
Structured (Complex): Explicitly requests a fact–reason–action schema, aligning closely with the benchmark annotation format.

Key Contributions

NORA Benchmark: A visual first-person video benchmark requiring models to generate candidate actions and justify them via explicit support graphs, moving beyond MCQs.
Grounded Reasonableness Metric: An evaluation protocol that distinguishes between recovering a broad space of reasonable actions and committing to a single well-supported action, using set-level and chosen-action metrics.
Empirical Analysis: A comprehensive benchmarking of 12 state-of-the-art VLMs (including GPT-5.x, Gemini, Qwen, and Grok families) revealing specific failure modes in normative reasoning.

Results

Benchmarking 12 VLMs reveals that while current models can often recover plausible actions and relevant scene facts, they consistently struggle with the full task of grounded reasonableness:

Performance Gap: Frontier models (GPT-5.2, GPT-5.1) achieve the highest scores ( $R \approx 0.38$ under Structured prompting) but remain significantly below the annotation-reference ceiling (GPT-5.4 reaches $R \approx 0.558$ ).
Component Analysis: Factual grounding is typically the strongest component, while action alignment (recovering the full reasonable action space) is the weakest. Models often describe the scene correctly but fail to construct the appropriate set of candidate next actions.
Prompt Effects: Structured prompting generally improves performance for high-capability models (e.g., GPT-5.2, Gemini-3-Flash) by providing scaffolding, but it can be a constraint for other models. However, at the chosen-action level, structured prompting consistently improves the final decision and its local grounding compared to deliberative prompting.
Set vs. Choice: There is a strong correlation between set-level reasonableness and chosen-action reasonableness, but they are distinct capabilities. Selecting one plausible action is easier than recovering the full reasonable action space.

Significance and Limitations

The paper claims that NORA shifts the evaluation question from "Can a model pick an action?" to "Can a model justify an appropriate action for the right visible reasons?" It provides a measurable gap in current VLMs' ability to perform situated normative reasoning, highlighting that current systems often lack the capacity to construct adequate option sets and bind them to local evidence.

Limitations acknowledged by the authors:

Annotation Pluralism: Human annotations are necessarily incomplete samples of the reasonable action space; multiple valid interpretations exist.
Comparative Scoring: Grounded reasonableness scores are intended for comparative diagnosis rather than as absolute measures of normative competence.
Bias: The benchmark may reflect biases from the source video distribution (Ego4D), the annotator pool, and the specific normative categories used.

The authors position NORA as both an evaluation framework and a potential training signal for embodied agents that must act reasonably in social environments, emphasizing that the current state of the art has substantial headroom for improvement in bridging the knowing–doing gap.