🤖 AI

SKG-VLA: Scene Knowledge Graph Priors for Structured Scene Semantics and Multimodal Reasoning for Decision Making

This paper introduces SKG-VLA, a multimodal decision-making framework that leverages Scene Knowledge Graphs to structure heterogeneous complaint evidence and policy knowledge, thereby enhancing reasoning accuracy, generalization, and robustness in large-scale complaint handling systems through a specialized three-stage training strategy.

Original authors: Zeyu Li, Lei Li

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zeyu Li, Lei Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to decide a complex legal case. You don't just have the plaintiff's story; you have a stack of evidence: photos of the damage, receipts, timestamps, chat logs, and a thick book of laws.

The Problem with Old Systems
Most current computer systems that handle customer complaints work like a very fast, but slightly clumsy, intern. They look at the customer's text, then they look at the photo, then they look at the order number, and they try to match these pieces to a pre-written template.

The Flaw: They treat these pieces of evidence as separate islands. They might see a photo of a broken box and a text saying "late delivery," but they struggle to connect the dots to realize, "Oh, the policy says if it's late and broken, we must refund, but only if the customer reported it within 24 hours." They often guess the answer based on surface-level keywords, leading to decisions that sound reasonable but break the rules.

The New Solution: SKG-VLA
The authors of this paper propose a new system called SKG-VLA. Instead of acting like an intern matching templates, this system acts like a detective building a case file.

Here is how it works, using simple analogies:

1. The "Scene Knowledge Graph" (The Detective's Whiteboard)

Instead of just reading text, the system takes the complaint and builds a Scene Knowledge Graph (SKG).

The Analogy: Imagine a detective's whiteboard covered in sticky notes.
- One note is the Customer's Story.
- One note is the Photo Evidence.
- One note is the Order History.
- One note is the Company Policy.
- The system draws strings connecting these notes. It connects "Late Delivery" to "Policy Rule 4" and "Broken Item" to "Refund Eligibility."
Why it helps: This creates a structured map of the situation. The system doesn't just see words; it sees the relationships between the story, the proof, and the rules. This prevents it from making decisions that contradict the company's own laws.

2. The "Training Camp" (Three-Stage Learning)

You can't just give a smart AI this whiteboard and expect it to work perfectly immediately. The authors trained the AI in three specific steps, like a student progressing through school:

Stage 1: Domain-Adaptive Pre-training (Learning the Vocabulary)
The AI reads thousands of complaint stories and learns the specific language of customer service, policies, and transaction terms. It's like reading the company handbook cover-to-cover before talking to a single customer.
Stage 2: Task-Oriented Instruction Fine-Tuning (Learning the Rules)
The AI practices solving specific problems using the "whiteboard" (the Graph). It learns to answer questions like, "Is the evidence enough?" or "Does this policy apply?" It learns to think in steps rather than just guessing.
Stage 3: End-to-End Multimodal Alignment (Seeing the Whole Picture)
Finally, the AI is connected to its "eyes." It learns to look at the actual screenshots and photos while looking at the text and the rules. It learns to say, "The text says the item arrived on time, but the photo shows a torn box, and the policy says torn boxes get a refund."

3. The "Synthetic Data Factory" (Practicing with Fake Cases)

To teach the AI, the authors didn't just use real complaints. They built a data synthesis pipeline.

The Analogy: Imagine a flight simulator. The system takes a real complaint and creates thousands of "what-if" scenarios.
- Scenario A: What if the photo was missing?
- Scenario B: What if the policy changed slightly?
- Scenario C: What if the customer reported it 3 days late instead of 1?
The AI practices making decisions on these simulated cases. This helps it handle rare, weird, or incomplete complaints that it might never have seen in real life.

The Results: Why It Matters

When they tested this system, it outperformed standard AI models in three key ways:

Rule-Following: It made fewer mistakes that violated company policies. It didn't just guess; it reasoned based on the "whiteboard."
Handling the Weird Stuff: It was much better at solving "long-tail" problems—those rare, complicated complaints that don't fit standard templates.
Resilience: When evidence was missing or blurry (like a bad photo or missing receipt), the system didn't panic. Because it understood the structure of the case, it could still make a logical decision based on what was there.

In Summary
The paper argues that to make good decisions on customer complaints, computers need to stop just "reading" and start "mapping." By turning a messy complaint into a structured map (the Scene Knowledge Graph) and training the AI to reason over that map, the system becomes a more reliable, fair, and accurate decision-maker.

Technical Summary of SKG-VLA: Scene Knowledge Graph Priors for Structured Scene Semantics and Multimodal Reasoning for Decision Making

1. Problem Definition

The paper addresses the limitations of current automated complaint handling systems in large-scale online service platforms. Existing systems typically rely on shallow classification or template matching over isolated modalities (e.g., text narratives, screenshots, metadata) and fail to utilize explicit scene structure, rule knowledge, and cross-evidence dependencies.

Complaint decision-making is inherently complex, requiring coordinated reasoning over heterogeneous evidence including:

User claims and narratives: The core complaint description.
Visual evidence: Screenshots, receipts, and UI captures.
Transactional context: Order records, timestamps, and service stages.
Policy constraints: Platform rules, eligibility conditions, and escalation criteria.
Historical interactions: Previous negotiation history.

The authors argue that without explicit structural representation, models often produce plausible-sounding but policy-inconsistent recommendations. The core challenge is to move from simple classification to multimodal reasoning over structured scene semantics to ensure decisions are grounded in policy, evidence sufficiency, and responsibility attribution.

2. Methodology: SKG-VLA Framework

The proposed solution, SKG-VLA, treats each complaint case as a "complaint scene" and utilizes a Scene Knowledge Graph (SKG) as the central representation. The methodology consists of three main components:

A. Scene Knowledge Graph (SKG) Representation
The SKG models a complaint case as a unified graph $G = (V, E, A)$ where:

Nodes ( $V$ ): Include complaint entities, evidence items, policy clauses, temporal events, transactional states, and decision nodes.
Relations ( $E$ ): Typed relations connecting the above entities.
Attributes ( $A$ ): Node-specific attributes.
The authors partition nodes into strongly coupled (critical for decision direction, e.g., order status, policy clauses) and weakly coupled (peripheral details) categories to guide graph generalization.

B. Data Synthesis Pipeline
To overcome the lack of datasets with explicit scene structure and auditable action supervision, the authors construct a large-scale dataset via a three-stage pipeline:

Generation: An LLM-based parser converts raw multimodal evidence ( $X = \{x_{text}, x_{img}, x_{meta}, x_{hist}, x_{rule}\}$ ) into an initial SKG ( $G_0$ ), structured scene description ( $D_0$ ), and QA pair ( $Q_0$ ).
Verification: An iterative loop verifies consistency between the graph, evidence, and policy, refining the outputs ( $G_k, D_k, Q_k$ ) until convergence.
Generalization: Rule-constrained graph edits ( $\Phi$ ) are applied to expand the scene space, generating rule-consistent variants that modify evidence sufficiency or service stages while preserving logical validity.

C. Three-Stage Training Strategy
The framework injects structured scene priors into a multimodal decision model through:

Domain-Adaptive Pre-training: Adapts the language model to complaint-domain semantics using generated scene descriptions and QA data.
Task-Oriented Instruction Fine-tuning: Trains the model to answer scene-related questions and output policy-grounded decisions.
End-to-End Multimodal Alignment: Connects the fine-tuned language model with a visual encoder and metadata projector to consume raw multimodal evidence directly.

3. Key Contributions

The paper outlines four primary contributions:

Reformulation of the Task: Defining multimodal complaint handling as a structured scene reasoning problem and introducing Scene Knowledge Graphs as explicit priors for understanding, policy grounding, and action recommendation.
Data Synthesis Framework: A pipeline that generates complaint scene descriptions, rule-consistent graph generalizations, QA supervision, and decision recommendations from multimodal evidence.
New Dataset and Benchmarks: Construction of a large-scale dataset (183,642 base cases expanded to 1.43M SKGs) and two in-domain evaluation suites:
- ComplaintScene-Text: A text-only benchmark focusing on policy-grounded reasoning.
- ComplaintScene-MM: A multimodal benchmark evaluating full scene reasoning under realistic evidence-rich settings.
Training Pipeline: A three-stage strategy that progressively transfers structured scene priors into a multimodal model, improving interpretability, robustness, and policy consistency.

4. Experimental Results

Experiments were conducted using Qwen2.5 backbones (0.5B, 1.5B, 3B) and compared against official general-purpose multimodal baselines (e.g., Qwen2.5-VL-3B-Instruct).

In-Domain Performance: SKG-VLA consistently outperformed baselines across all model scales. On the ComplaintScene-MM benchmark, the 3B SKG-VLA model achieved an average score of 84.48, surpassing the official Qwen2.5-VL-3B-Instruct (80.88) and the multimodal baseline without SKG priors (82.17). Gains were most significant in Responsibility Attribution and Resolution Prediction, the most complex decision tasks.
Policy Grounding: The model showed superior Policy Consistency (84.02% for 3B SKG-VLA vs. 75.61% for the generic baseline), indicating better adherence to domain rules.
Generalization: SKG-VLA demonstrated stronger performance on Rare-Type complaint scenarios (77.46% vs. 69.84% for the baseline), suggesting the structured priors help the model extrapolate beyond frequent templates.
Robustness: Under evidence corruption (30% missing data), SKG-VLA degraded more gracefully than baselines, maintaining higher accuracy. This indicates that decisions are stabilized by graph-level semantics rather than relying solely on local visual cues.
Transfer Learning: While the generic baseline retained a slight edge on broad transfer tasks (SIMMC 2.0, DocVQA), SKG-VLA remained competitive while significantly outperforming on complaint-specific tasks.

Ablation Studies confirmed that the explicit SKG graph structure provided the largest performance gain, followed by policy nodes and rule-consistent graph generalization.

5. Significance and Claims

The paper claims that SKG-VLA addresses a critical gap in automated decision support by moving beyond surface-level matching to structured scene reasoning.

Policy Consistency: The primary significance lies in the ability to generate decisions that are auditable and strictly consistent with platform policies, a requirement often missed by general-purpose multimodal models.
Handling Incomplete Evidence: The framework is particularly effective in "long-tail" and "incomplete evidence" scenarios, which are common in real-world operational systems where screenshots may be missing or metadata incomplete. The SKG provides a structural scaffold that allows the model to reason even when specific evidence is absent.
Interpretability: By organizing entities and relations into a graph, the system offers a more interpretable reasoning path compared to black-box classification, facilitating better human oversight.

The authors conclude that while general multimodal models are powerful, they require domain-specific structural priors (like SKGs) to handle the rule-constrained, evidence-dependent nature of complaint decision-making effectively.