The methodology of Constructing the Large-Scale Dataset for Detecting Presuicidal and Anti-Suicidal Signals in Social Media Texts in Russian

Imagine the internet as a giant, bustling digital town square where millions of people share their thoughts, jokes, and deepest secrets every second. In this noisy crowd, some people are quietly screaming for help, dropping subtle hints about feeling hopeless or thinking about ending their lives. For years, scientists have been trying to build "digital detectives"—computer programs that can read these social media posts and spot the signs of danger before it's too late. This field, known as Natural Language Processing (NLP), is like teaching a robot to understand human emotions by reading their words. But here's the tricky part: computers are terrible at guessing feelings unless they are trained on a massive library of examples that humans have already labeled as "sad," "angry," or "in danger." Without these carefully curated libraries, the computer is just guessing in the dark.

This is exactly the challenge tackled by a team of researchers from Russia who decided to build a massive, specialized library for a very serious job: finding signals of suicide risk in Russian social media. They didn't just want to find people who are sad; they wanted to distinguish between those who are spiraling toward self-harm (presuicidal signals) and those who are finding reasons to keep going (anti-suicidal signals). Think of it like sorting a giant pile of mixed-up mail: some letters are urgent distress calls, some are hopeful notes about the future, and most are just regular junk mail. The researchers spent years creating a strict rulebook, training a team of human "sorters," and building a dataset of over 57,000 text examples to teach computers how to tell the difference. Their work suggests that while the task is incredibly difficult and subjective, a well-structured dataset can help computers get much better at spotting these life-saving signals.

The Mission: Finding the Needle in the Haystack

The researchers started with a simple but terrifying reality: suicide is a major problem, and many people who are struggling write about it online before they act. However, these cries for help are buried under mountains of irrelevant posts—memes, complaints about the weather, and random thoughts. To help humans find these people faster, the team needed a way to train computers to filter the noise.

They set out to create a "Rosetta Stone" for suicidal texts in Russian. Their goal wasn't just to say "this person is sad," but to categorize the specific reasons why. Is the person feeling hopeless? Are they being bullied? Do they have a plan? Or, conversely, are they talking about their love for their family or their plans for tomorrow? These are the "anti-suicidal signals"—the reasons to stay.

Building the Rulebook: The Instruction Manual

Before they could train a computer, they had to teach the humans who would label the data. This was the most delicate part of the project. The researchers realized that asking people to read thousands of depressing posts could be emotionally draining, so they built a safety net. They created a detailed instruction manual that acted like a map for the annotators (the people doing the labeling).

This manual included:

• A "Mood Board" of Classes: Instead of just "sad" or "not sad," they created 33 specific categories for danger signals (like "thoughts about death," "alcohol problems," or "family breakdown") and 12 categories for hope signals (like "expression of love" or "positive self-esteem").
• The "First-Person" Rule: A crucial rule was that the text must be about the author. If someone writes, "My friend wants to die," that's irrelevant. If they write, "I want to die," that's a signal. This rule was strict to avoid false alarms.
• Emotional Check-ins: Before and after every batch of work, the annotators had to take tests to check their own mental health. If they felt too overwhelmed, they were told to stop immediately.

The Human Element: Sorting the Chaos

The team recruited a group of annotators, including both machine learning experts and regular people, to label over 57,000 text examples. They didn't just throw the data at them; they used a smart sampling strategy. They broke the data into chunks and used preliminary computer models to guess which texts might be interesting, ensuring they got a good mix of different types of posts.

The process was messy and human. The researchers found that even with a strict rulebook, people disagreed. One person might see a metaphor as a cry for help, while another sees it as just a figure of speech. To handle this, they used a "soft majority voting" system. If three people labeled a post, and two agreed on a specific danger signal, that label stuck. This approach allowed them to keep the data rich and varied without getting stuck on every single disagreement.

They also discovered that the "irrelevant" posts were the hardest to handle. Sometimes, a post sounded dangerous but was actually just a story about a character in a book or a quote from a movie. The team had to go back and re-label thousands of these "uncertain" examples, refining their rules to make sure the computer didn't get confused by third-party stories.

The Results: Teaching the Computer

Once the data was cleaned and labeled, the team trained a computer model (specifically a model called RuBERT) to read the texts. They tested the model in different ways:

• The "Big Picture" Test: Could the model tell the difference between "danger," "hope," and "nothing"? Yes, and it was quite good at this, achieving a score of about 0.71.
• The "Fine Detail" Test: Could the model tell the difference between "feeling guilty" and "feeling hopeless"? This was harder. The model struggled more when asked to be too specific, suggesting that it's easier for computers to spot the general mood than the exact nuance.

The experiments showed that the more specific the categories, the harder it was for the model to learn. However, the model performed best when it was allowed to group similar feelings together. This suggests that while computers can learn to spot the general signs of danger, the fine details of human emotion are still a bit of a mystery to them.

What This Means

The paper doesn't claim to have solved the problem of suicide detection. Instead, it offers a powerful tool: a massive, carefully constructed dataset and a proven method for creating one. The researchers found that the biggest hurdles were the "gray areas"—texts that were ambiguous, metaphorical, or about other people. They also highlighted that the "anti-suicidal" signals (the reasons to live) were harder for the model to learn than the danger signals, likely because the categories for hope were a bit too abstract.

In the end, this work is a foundation. It's a giant, open library of Russian social media posts that anyone can use to build better tools. The authors suggest that future work could use even smarter AI to help with the labeling and that they need to keep refining the rules to handle the tricky, subjective parts of human language. They have made their data, code, and instructions public, inviting others to join the effort to build a digital safety net that is more accurate, more empathetic, and ultimately, more helpful.