Technical Summary: Falcon – Functional Assembly and Language for Compositional Reasoning in X-ray

1. Problem Formulation: Compositional Threat Reasoning

Current vision-language models (VLMs) and X-ray security systems are predominantly object-centric, focusing on detecting and describing individual entities (e.g., a gun or a knife). However, in safety-critical X-ray baggage screening, threats often arise not from single objects but from the functional compatibility of spatially dispersed components. Improvised Explosive Devices (IEDs) may be transported in a dismantled state, with batteries, detonators, and explosive charges separated within a bag. Individually benign, these components become hazardous only when functionally compatible.

The paper identifies a gap in existing research: while large-scale benchmarks exist for object detection, they lack the ability to model relational safety inference. Current Multimodal Large Language Models (MLLMs) rely on implicit attention without explicit supervision over inter-component compatibility, making it difficult to reason about whether spatially separated components under heavy occlusion collectively constitute a functional threat.

The authors formalize this setting as Compositional Threat Reasoning, where safety is modeled as a relational property of grounded regions rather than an independent detection outcome.

2. Methodology: The Falcon Framework

The authors propose Falcon, a segmentation-aware multimodal framework designed to abstract region features into a structured safety state before language generation. The architecture enforces relational consistency between perception and reasoning through an explicit intermediate interface.

2.1 Architecture Overview

Falcon integrates four key components:

1. Segmentation-Aware Perception:

   • Visual Backbone: Uses DINOv2 to extract dense patch tokens, preserving spatial resolution critical for small, cluttered components.
   • Region Proposer: Employs a class-agnostic transformer (RF-DETR) to generate instance proposals, including bounding boxes, segmentation masks, and confidence scores. Mask prediction is prioritized over bounding boxes to handle X-ray superposition and clutter.
   • Mask-Aware Region Encoder: Combines ROI-aligned features with mask-pooled features to create language-aligned region embeddings.

2. Structured Functional State Modeling (Structured Safety Adapter - SSA):

   • The SSA maps variable-sized region embeddings into a fixed, interpretable functional state.
   • Query-Based Component Slots: Uses learned queries corresponding to a predefined component taxonomy (e.g., battery, detonator, main charge) to aggregate region embeddings via attention.
   • Structured Predictions: The SSA predicts three specific signals:
     • Component Presence ($\hat{p}$): Binary probability of each component type.
     • Pairwise Functional Links ($\hat{L}$): Probabilities indicating the functional compatibility between component pairs (e.g., battery-to-detonator).
     • Scene Risk ($\hat{r}$): A continuous risk score ($0$ to $1$) derived from the structured component embeddings and inferred links, rather than raw visual tokens.

3. Tokenized Safety Conditioning:

   • The structured predictions are converted into "safety tokens" and injected into the Large Language Model (LLM) as an explicit intermediate representation. This conditions the LLM on the inferred safety state before it generates a response.

4. LLM Reasoning:

   • Uses Vicuna-7B as the backbone. The model fuses image tokens, region tokens, safety tokens, and text tokens. The LLM is frozen and adapted using LoRA for domain-specific fine-tuning.

2.2 Training Strategy

Falcon is trained in three decoupled stages:

1. Stage 1 (Proposal Learning): Trains the RF-DETR detector in a class-agnostic manner to produce high-quality instance masks and boxes.
2. Stage 2 (Structured Multimodal Alignment): Freezes the detector and LLM. Trains the SSA to predict presence, links, and risk, while jointly supervising language generation and structured safety prediction.
3. Stage 3 (Instruction Fine-Tuning): Enables LoRA adapters in the LLM to improve instruction adherence and alignment between the structured predictions and generated explanations.

3. The Falcon-X Benchmark

To evaluate this problem, the authors introduce Falcon-X, a benchmark of approximately 7,000 real dual-energy X-ray scans (expanded to ~50,000 with counterfactuals) featuring inert, dismantled IED components.

• Unique Features: Unlike existing datasets (e.g., SIXray, OPIXray) that focus on intact prohibited items, Falcon-X models spatially dispersed functional components.
• Annotations: Includes dense instance grounding (bounding boxes and pixel-accurate masks) and structured supervision for component completeness and calibrated risk.
• Task Suite: The benchmark evaluates three hierarchical layers:
  1. Grounded Perception: Scene captioning, panoptic segmentation, and referring segmentation under heavy occlusion.
  2. Compositional Functional Reasoning: Missing component identification, functional completeness scoring, and referring functional grounding.
  3. Relationally Consistent Safety Inference: Scene risk prediction, functional link estimation, and logical consistency checks.

4. Experimental Results

Experiments compare Falcon against general-purpose and domain-adapted VLMs (e.g., LISA, Groma, Sting-Bee) on Falcon-X.

• Scene Understanding & VQA: Falcon performs on par with or slightly exceeds strong baselines in standard captioning and VQA, indicating that the structured approach does not degrade general multimodal capabilities.
• Compositional Grounding: Falcon demonstrates a significant advantage in Referring Functional Grounding (RFG), where the target is defined by functional composition rather than appearance. It outperforms the next best baseline by +30.38 points in cIoU and +40.03 points in mIoU. This suggests that relational supervision is essential for identifying components that form a threat, even if they are not visually distinct as a single object.
• Safety Inference: While component presence detection saturates across models (approx. 98%), Falcon significantly reduces the Mean Absolute Error (MAE) in Scene Risk Level and Component Link Risk estimation. This confirms that the structured modeling of component interactions improves the calibration of threat assessments.
• Ablation Studies:
  • Removing the link prediction head causes the largest performance drop, confirming that modeling relational compatibility is the primary driver of functional grounding.
  • Oracle ablations show that while perception is a bottleneck, the reasoning module (SSA) provides substantial headroom for improvement, validating the efficacy of the structured state injection.

5. Significance and Claims

The paper claims that Falcon establishes compositional safety reasoning as a distinct evaluation paradigm for multimodal systems. Its primary contributions are:

1. Formalization: It defines threat assessment in X-ray imagery not as object detection, but as a relational property of grounded components.
2. Benchmarking: Falcon-X is the first dataset to jointly support dense grounding, multimodal understanding, and structured functional threat reasoning, specifically addressing the challenge of dismantled components.
3. Methodological Advance: By injecting an explicit structured safety state (presence, links, risk) into the LLM, Falcon moves beyond implicit attention mechanisms to enforce relationally consistent and safety-aware reasoning.

The authors conclude that while existing models adapt well to the appearance of X-ray images, they struggle with the functional compatibility of dispersed components. Falcon addresses this by transforming threat assessment from an implicit attention-based process into a structured inference problem, offering a principled approach for safety-critical analysis in complex visual environments.