Technical Summary: REASON – A Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition

Problem Statement

Skeleton-based Human Activity Recognition (HAR) has achieved strong empirical performance using deep learning models, particularly Graph Convolutional Networks (GCNs). However, these models function as "black boxes," lacking interpretability. This opacity is problematic for high-stakes domains like healthcare and rehabilitation, where transparent decision-making is essential. Furthermore, existing neurosymbolic approaches face fundamental integration challenges:

1. Symbol Grounding: Bridging continuous joint trajectories with discrete symbolic predicates is difficult. Traditional heuristics (e.g., "knee angle < 90°") are brittle and fail under pose variations or viewpoint changes.
2. Temporal Dynamics: Actions are inherently temporal. Purely spatial logic cannot distinguish actions with similar static poses but different execution orders or motion directions (e.g., Wearing Glasses vs. Taking Off Glasses).
3. End-to-End Learning: Discrete Boolean operators typically block gradient propagation, preventing the joint optimization of perception (feature extraction) and reasoning (rule application).

Methodology: The REASON Framework

The authors propose REASON (Rule-based Explainable Action via Symbolic cONcepts), a neurosymbolic framework that reframes action recognition as concept-driven first-order logical reasoning over learnable motion primitives. The framework operates through four key stages:

1. Concept Bank Generation

Instead of relying on manual heuristics, the authors construct a concept bank $\mathcal{C}$ comprising spatial, temporal, and interactive motion primitives using a three-stage Large Language Model (LLM) pipeline:

• Description Generation: An LLM (GPT-3) decomposes actions into fine-grained body-part descriptions (e.g., "Leg: bends at knees then pushes explosively").
• Abstraction & Clustering: Raw descriptions are converted into canonical concepts by extracting verb-direction-modifier patterns, embedding them, and clustering them via k-means. This yields spatial concepts (e.g., arm raise, leg jump).
• Temporal & Interaction Definition: Explicit temporal concepts are defined to capture motion direction (motion upward), sequence (hands first), and dynamics (speed fast). An interaction concept handles mutual actions.
• Association: A second LLM (LLaMA-3.2) matches actions to the concept bank, creating a binary association matrix $M$ where $M_{a,c}=1$ if concept $c$ is active for action $a$. The final bank contains 74 concepts (52 spatial, 21 temporal, 1 interaction).

2. Skeleton Representation Learning

A standard spatio-temporal skeleton encoder (e.g., Hyper-GCN) processes input sequences $X \in \mathbb{R}^{T \times V \times 3}$ without global pooling to preserve fine-grained motion information. The encoder feeds into three branches:

• STC-Decoder: Predicts concept activations.
• Text Alignment: Aligns pooled skeleton features with concept-grounded text embeddings (generated by a CLIP encoder with LoRA fine-tuning) using a contrastive loss ($L_{align}$). This ensures the skeleton features cluster with their semantic descriptions.
• Part Divergence: A loss ($L_{div}$) penalizes high cosine similarity between features of different body parts, ensuring discriminative representations for each anatomical region.

3. Spatio-Temporal Concept Decoder (STC-Decoder)

This module bridges continuous features and symbolic predicates. It decouples encoder features into spatial ($F_s$) and temporal ($F_t$) streams.

• Group-Decoding: To handle the complexity of heterogeneous concept relationships, concepts are partitioned into groups. Learnable query matrices perform cross-attention to selectively aggregate skeleton information relevant to specific concept subsets.
• Prediction: A Group-FC layer projects these queries into soft concept probabilities $\hat{c} \in [0, 1]^{|\mathcal{C}|}$.

4. Differentiable First-Order Logic Reasoning

To enable end-to-end training while maintaining discrete interpretability:

• Binarization: Soft predictions are binarized ($\bar{c} = \mathbb{I}[\hat{c} > 0.5]$) using a Straight-Through Estimator (STE) to allow gradient flow.
• Logical Layers: Stacked layers of conjunction (AND) and disjunction (OR) compose concepts into rules.
• Gradient Grafting: During the forward pass, discrete operations and weights are used for inference. During backpropagation, gradients are computed through continuous relaxations (soft AND/OR) and "grafted" onto the discrete path, enabling the learning of sparse, interpretable logical rules.
• Classification: A linear classifier maps rule activations to final action predictions.

Key Contributions

1. REASON Framework: A novel neurosymbolic formulation for skeleton-based HAR that integrates concept-grounded representations with differentiable first-order logic for end-to-end training.
2. STC-Decoder: A specialized decoder that independently extracts spatial body-configuration and temporal motion-dynamics concepts, providing robust abstractions against execution variability.
3. Concept Bank: A curated bank of 74 spatial and dynamic motion concepts for NTU RGB+D and NW-UCLA datasets, generated via LLMs to ensure semantic richness without manual enumeration.
4. Interpretability: The system provides both local interpretability (instance-specific concept activations) and global interpretability (human-readable logical rules governing action semantics).

Experimental Results

The framework was evaluated on NTU RGB+D 60/120 and NW-UCLA benchmarks.

• Performance: REASON achieves competitive performance, surpassing several black-box baselines.
  • On NTU RGB+D 60 (X-Sub), it achieves 94.3% accuracy (4-stream), outperforming strong GCN baselines like Hyper-GCN (93.7%) and BlockGCN (93.1%).
  • On NTU RGB+D 120 (X-Set), it reaches 91.9%.
  • On NW-UCLA, it achieves 97.9%.
• LLM Re-ranking: An optional re-ranking stage using LLaMA-3.2 7B grounded in concept activations further boosts performance to 95.6% (NTU-60 X-Sub) and 93.2% (NTU-120 X-Set), outperforming other LLM-augmented methods that rely on opaque features.
• Ablation Studies:
  • Temporal Concepts: Adding temporal concepts to spatial ones improved accuracy by +7.8%, confirming their necessity for distinguishing similar poses.
  • Logic Layer Capacity: A 128-node configuration was optimal; disabling negation caused a catastrophic drop to 49.87%, highlighting the need for $\neg$ concepts (e.g., distinguishing standing from walking via $\neg$ leg walk).
  • Concept Intervention: Correcting mispredicted concepts in the logic chain monotonically improved accuracy (up to +5.67% for 3 corrections), proving that errors stem primarily from concept misprediction rather than flawed logical rules.

Significance and Claims

The paper claims that neurosymbolic reasoning is an effective paradigm for interpretable spatio-temporal action understanding. By grounding skeleton representations in learnable motion concepts and composing them via differentiable logic, REASON achieves state-of-the-art performance while maintaining fully transparent concept-rule reasoning chains.

Unlike prior work that relies on post-hoc explanations or brittle hand-crafted rules, REASON learns action-defining rules directly from data. The authors emphasize that this approach enables systematic analysis of both correct classifications and failure cases, offering a pathway toward trustworthy AI in safety-critical applications where the "why" behind a prediction is as important as the prediction itself. The framework demonstrates that it is possible to bridge the gap between neural perception and symbolic inference without sacrificing accuracy or scalability.