Technical Summary: TSAI-MetaFraud

Problem Statement

The rapid evolution of metaverse platforms has established persistent digital ecosystems where users engage in social interaction, virtual asset trading, and complex economic activities. While these environments offer new opportunities, they introduce significant security challenges, including fraudulent transactions, automated bot activity, account manipulation, and coordinated malicious behavior.

Existing research in fraud detection has largely relied on datasets from traditional domains such as banking, e-commerce, and cryptocurrency. However, these resources typically focus on isolated aspects—either transaction networks or user behavior—rather than the unified, multimodal nature of metaverse economies. Furthermore, publicly available metaverse datasets often lack critical components such as financial transaction records, fraud annotations, or bot labels simultaneously. This scarcity prevents the development and reproducible evaluation of multimodal fraud detection methods that can jointly analyze behavioral activity, financial interactions, and relational structures within a single virtual economy.

Methodology

To address these limitations, the authors introduce TSAI-MetaFraud, a multimodal, multi-task benchmark dataset generated within a simulated metaverse environment.

Data Generation and Environment

• Platform: The dataset was constructed using OpenSimulator, an open-source platform compatible with the Second Life protocol, which supports avatar interactions, virtual asset ownership, and in-world currency transfers.
• Population: The simulation involved 936 active avatars representing diverse behavioral and financial profiles.
  • Benign Users: Performed normal movements and legitimate transactions.
  • Behavioral Fraud: Automated bots exhibiting synthetic interaction patterns and abnormal keystroke characteristics.
  • Financial Fraud: Accounts with realistic movement behaviors but engaging in illicit financial activities (e.g., transaction layering).
  • Hybrid Fraud: Accounts combining automated behavioral patterns with fraudulent financial activity.
  • Unknown: A subset of accounts with masked labels to simulate real-world investigation uncertainty.
• Behavioral Modeling: Human-like behavior was modeled using keystroke dynamics derived from the public KMT dataset, fitted with Gaussian Kernel Density Estimators (KDEs). Automated accounts were generated using "naive" bots (rigid timing) and "stealth" bots (sampling human distributions with reduced variance) to create meaningful overlap between benign and malicious classes.
• Data Collection: A multi-layer pipeline captured movement trajectories, object interactions, and keystroke timing in real-time, alongside server-side logs for all financial events (timestamps, amounts, fees, types).

Data Representation

The dataset is distributed in two formats to support diverse research communities:

1. Graph Format: A modular relational graph representing avatars and sessions as nodes and transactions as directed edges. It includes four tables: avatar attributes, session biometrics, transaction edges, and avatar labels.
2. Tabular Format: A unified flat representation where each transaction edge is a 24-dimensional feature vector merging transaction attributes, sender/receiver network centralities, and sender session biometrics.

Benchmark Tasks

The authors define four specific tasks to evaluate multimodal and graph-based methods:

1. Transaction Fraud Detection: Multiclass classification of transaction edges (Benign, Behavioral Fraud, Financial Fraud, Hybrid) under a strict inductive setting (unseen accounts).
2. Cross-Modal Node Classification: Classifying avatar nodes into four categories using combined session-level behavioral features and graph-derived account characteristics.
3. Temporal Link Prediction: Forecasting future transactions in an evolving network based on historical graph snapshots.
4. Weakly Supervised Fraud Detection: Inferring multiclass labels for a largely unlabeled population (90% labels masked) to simulate realistic investigation constraints.

Key Contributions

1. TSAI-MetaFraud Dataset: The first multimodal benchmark that jointly captures behavioral activity, financial transactions, graph-structured interactions, and fraud-related behaviors within a unified metaverse environment.
2. Comprehensive Benchmark Suite: Definition of four distinct tasks spanning fraud detection, bot detection, temporal link prediction, and weakly supervised learning, enabling systematic evaluation of multimodal methods.
3. Baseline Evaluations: Provision of baseline results using classical machine learning models (Logistic Regression, Random Forest, XGBoost) and Graph Neural Networks (GraphSAGE, Label Propagation) to characterize the dataset's difficulty and establish reference performance levels.
4. Open Release: Public availability of the dataset and evaluation framework to support reproducible research.

Results and Analysis

Baseline experiments reveal significant challenges and distinct performance characteristics across modalities:

• Behavioral vs. Financial Fraud: Tree-based models (Random Forest, XGBoost) achieved strong performance in detecting Behavioral Fraud (automated scripts) and benign users, leveraging session-level features like activity intensity and timing. However, these models completely failed to identify Financial Fraud and Hybrid fraud (F1-scores of 0.00), indicating that financial illicit patterns are expressed through transaction structures rather than local behavioral attributes.
• Graph Neural Networks: The heterogeneous GraphSAGE model demonstrated the ability to detect Financial and Hybrid fraud by leveraging transaction relationships and network structure. However, it struggled with Behavioral Fraud compared to tree-based methods, suggesting that standard message-passing mechanisms may smooth out the high-frequency local anomalies (e.g., keystroke timing) that define automated behavior.
• Temporal Link Prediction: Traditional topological heuristics (Common Neighbors, Jaccard Coefficient) performed poorly. GraphSAGE significantly outperformed these methods, highlighting the necessity of learned structural representations for dynamic networks.
• Weakly Supervised Learning: Under severe label scarcity (90% masked), all methods experienced substantial performance degradation. Graph diffusion methods (Label Propagation, GraphSAGE) struggled to propagate signals from the few available labeled nodes for rare classes, resulting in low macro F1-scores (~0.23).

Significance and Claims

The authors position TSAI-MetaFraud as a critical resource for advancing multimodal learning, graph mining, and trustworthy AI in emerging metaverse ecosystems.

• Bridging the Gap: The dataset addresses the lack of realistic benchmarks that unify user behavior, financial activity, and interaction networks, which is essential for developing robust fraud detection systems.
• Multimodal Necessity: The results explicitly demonstrate that behavioral and transactional modalities provide complementary information. Behavioral features are essential for bot detection, while graph structure is indispensable for identifying complex money-laundering patterns.
• Realistic Constraints: By incorporating class imbalance, temporal evolution, and limited supervision, the benchmark reflects the practical difficulties of fraud investigation, motivating the development of more robust, label-efficient, and adaptive learning methods.

The paper concludes that TSAI-MetaFraud provides a rigorous testbed for evaluating future approaches in virtual-world security, offering a unified platform for reproducible research in an application domain that is rapidly evolving yet currently lacks standardized evaluation resources.