Technical Summary: An Evaluation of Chat Safety Moderations in Roblox

1. Problem Statement

Roblox, a massively multiplayer online (MMO) platform with nearly 144 million daily active users, hosts a significant population of underage players. While the platform employs automated real-time chat moderation to filter inappropriate content, there is a lack of independent, large-scale investigation into the effectiveness of these systems. Concerns persist regarding the platform's ability to detect nuanced, evolving harmful behaviors such as grooming, sexual exploitation, bullying, and off-platform pivoting. Unlike explicit profanity, these threats often develop slowly, vary in style, and rely on context, making them difficult for keyword-based or isolated message-level filters to detect. Furthermore, users may actively employ evasion strategies to bypass these filters. The authors aim to scientifically evaluate Roblox's chat safety moderation by answering two primary research questions:

• RQ1: What child-safety violations are bypassing Roblox's chat moderation system?
• RQ2: What strategies are policy-violating users employing to intentionally bypass the chat moderation system?

2. Methodology

Data Collection Framework

Due to the absence of a public API for user content and the ephemeral nature of chat messages in Roblox, the authors developed a non-intrusive, Optical Character Recognition (OCR)-based framework to collect data from public game servers.

• Game Selection: Four popular games were selected: Brookhaven RP, Adopt Me!, Royal High, and Berry Avenue RP. Data was collected from both 9+ and 13+ age settings.
• Recording: Screen recordings were captured at 60 FPS from public servers over seven consecutive days per game/age combination, totaling 336 hours of footage.
• Preprocessing: To handle dynamic gameplay backgrounds that interfere with OCR, the authors utilized a metamorphic testing approach. They generated six image variants (grayscale, Gaussian blur, Otsu binarization, etc.) for each frame. If confidence remained low, they applied background suppression.
• Transcription: A hybrid approach was used. High-confidence whole-frame OCR was attempted first. If confidence dropped below 95%, a segmentation-based approach localized text boxes for line-level OCR.
• Validation: The framework achieved a 91.76% recall and 97.44% Average Matched Similarity (AMS) against a manually transcribed ground truth dataset, significantly outperforming single-method approaches.
• Corpus: The final dataset comprises approximately 2.04 million chat messages from 105,214 users. All Personally Identifiable Information (PII) was redacted.

Analysis Pipeline (RQ1: Escaped Categories)

Given the scale of the dataset, manual qualitative analysis was infeasible. The authors adopted a two-step, semi-automated approach:

1. LLM Pre-filtering: A locally hosted Large Language Model (LLM), gpt-oss-120b, was fine-tuned with few-shot prompting on a ground truth sample of 2,000 conversations. This model achieved a recall of 71.43% and precision of 31.39% on the ground truth. It was applied to the full corpus to identify potentially unsafe conversations.
2. Thematic Saturation Analysis: The model flagged 7,254 conversations as unsafe. Researchers then performed iterative open and axial coding on these conversations until thematic saturation was reached (no new patterns emerged for 5 consecutive iterations). This process involved manual verification to distinguish true positives from false positives.

Analysis Pipeline (RQ2: Evasion Techniques)

To study evasion, the authors focused on users whose messages had been previously flagged (masked with ### by Roblox).

• Detection: They identified masked sequences using rule-based heuristics (length, character composition) to distinguish moderation hashes from natural text.
• Sampling: A stratified sampling process was used to select 94 users based on their frequency of masked messages (High, Medium, Low frequency groups).
• Analysis: 12,612 messages from these users were manually reviewed to develop a taxonomy of evasion behaviors, stopping at thematic saturation.

3. Key Contributions

1. Non-Intrusive Data Collection: Introduction of an OCR-based framework capable of extracting chat data from video recordings of gaming environments where textual APIs are unavailable.
2. Scalable Qualitative Analysis: Development of a saturation-based thematic analysis approach augmented by LLM pre-filtering to extract insights from a corpus of ~2 million messages.
3. Empirical Audit of Moderation: The first large-scale characterization of moderation failures in Roblox, identifying specific harmful patterns that escape detection and the specific linguistic tactics users employ to evade them.

4. Key Findings

RQ1: Escaped Harmful Categories

The study found that while Roblox's system filters isolated explicit terms, it frequently fails to detect unsafe content that emerges over the course of a conversation. The dominant categories of escaped content included:

• Grooming: The largest category, involving trust-building, requests for off-platform contact, age/gender probing, and solicitation of explicit images or in-person meetings.
• Bullying & Harassment: Including racial slurs, identity-based degradation, and profane abuse, often using coded language to bypass filters.
• Sexual Content: Ranging from solicitation and sexting to objectification and sexualized rumors, often conveyed through euphemisms or fragmented spelling.
• Threats & Violence: Including descriptions of physical harm, forced violence in roleplay, and threats of self-harm.
• PII Leakage: Sharing or requesting off-platform handles (e.g., TikTok, Discord) and sensitive personal information.
• Profanity: Use of altered spellings (e.g., "f4", "btx") to bypass keyword filters.

RQ2: Evasion Strategies

Users who were previously moderated demonstrated a sustained adversarial cycle, employing specific techniques to bypass detection:

• Multi-Line Fragmentation: Distributing a single harmful intent across multiple consecutive chat turns (e.g., splitting a sentence so individual lines appear benign).
• Lexical Retry: Reformulating blocked content by changing word order or spelling while preserving the underlying intent.
• Altered Spelling: Using phonetic spellings, added punctuation, or character substitutions (e.g., "shi." instead of "shit") at the word level.
• Code Words: Using abbreviations or context-dependent substitutes (e.g., "BTC" for "bitch") that rely on shared interpretation.
• Leet Speak: Replacing letters with visually similar numbers or symbols (e.g., "n@me", "d8ting").
• Probing: Explicitly testing the system's boundaries by attempting variations of a blocked term (e.g., describing an app by its start and end letters) to learn what is allowed.

5. Significance and Limitations

Significance:
The paper provides the first empirical evidence that Roblox's current moderation pipeline is insufficient for detecting context-dependent harms like grooming and evasion tactics. The authors argue that the current message-level moderation fails to capture the trajectory of harmful interactions. They highlight that a small subset of persistent users is responsible for the bulk of harmful content, suggesting that moderation strategies need to shift from isolated message filtering to user-level behavioral analysis.

Limitations:

• Scope: Data was collected only from public servers; private servers were not included.
• Language: The study primarily focused on English; non-English languages and emoticons introduced noise in transcription.
• Model Performance: The pre-filtering LLM had a recall of ~71%, meaning the study likely underestimates the true prevalence of unsafe interactions (a lower-bound estimate).
• Ethical Constraints: The use of local models and anonymization was strictly enforced, but the reliance on video recording and OCR introduces potential for minor transcription errors compared to direct API data.

Recommendations:
The authors propose a shift toward a user-centered, layered moderation pipeline:

1. Hybrid Detection: Combining deterministic methods (regex, string matching) for known evasion patterns with probabilistic models for complex cases like grooming.
2. Contextual Analysis: Evaluating conversations as a whole rather than isolated messages to detect intent progression.
3. Longitudinal Monitoring: Applying stricter, more sophisticated moderation to users with a history of flagged messages across different games.
4. Trust Building: Improving reporting mechanisms to ensure users believe their reports lead to actionable safety improvements.