🤖 machine learning

Towards Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects

This survey systematically reviews and synthesizes self-supervised learning methodologies for event stream modeling across diverse domains, proposing a unified taxonomy and future research agenda to overcome data scarcity and fragmented efforts by bridging isolated domain-specific approaches.

Original authors: Levente Zólyomi, Tianze Wang, Sofiane Ennadir, Oleg Smirnov, Lele Cao

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Levente Zólyomi, Tianze Wang, Sofiane Ennadir, Oleg Smirnov, Lele Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, the digital world generates a relentless river of moments. When a patient visits a clinic, a shopper clicks on a product, a player moves a character in a game, or a bank processes a transaction, a record is created. These records are not just static lists; they are continuous, time-stamped sequences of events that tell a story about how a person or system behaves over time. In fields ranging from medicine to finance, this stream of data holds the key to understanding complex patterns, predicting future needs, and making better decisions. However, a significant hurdle blocks the path to unlocking this potential: most of these records come without labels. Unlike a textbook with answers at the back, these event streams rarely tell us what the data means or what outcome it predicts. To make sense of them, computers traditionally need vast amounts of human-labeled examples, which are expensive, slow to produce, and often impossible to get due to privacy laws.

To overcome this shortage of labeled data, researchers have turned to a method called self-supervised learning. Imagine a student who learns a language not by memorizing a dictionary of definitions, but by reading thousands of books and figuring out the rules of grammar and meaning on their own. Self-supervised learning works similarly for machines. It allows computers to study the raw, unlabeled flow of events and teach themselves to recognize patterns, relationships, and structures without needing a human to point out what is important. This approach has revolutionized how machines understand text and images, but its application to these time-based event streams has been scattered and inconsistent.

A new survey by Levente Zólyomi and his colleagues brings these scattered efforts together, offering a unified view of how self-supervised learning is being used to model event streams across different industries. The researchers reviewed hundreds of studies from healthcare, e-commerce, gaming, and finance to see how scientists are teaching machines to learn from unlabeled data. They found that while the field is rich with innovation, it is currently fragmented, with experts in one industry often reinventing methods that their counterparts in another industry have already discovered. The paper maps out the current landscape, identifying which techniques work best for specific types of tasks and highlighting the gaps that need to be filled before these tools can be used reliably everywhere.

The core of the survey reveals two main ways researchers are currently teaching machines to learn from these event streams. The first approach, which dominates the current literature, is predictive. In this method, the computer is given a sequence of events with some parts hidden or removed, and it must guess what is missing. For example, in a healthcare setting, if a patient's medical history has a gap in the record, the model tries to predict what diagnosis or treatment likely occurred during that time. This is similar to how a person might fill in a missing word in a sentence based on the context of the surrounding words. This technique has proven highly effective for tasks that require understanding the full context of a sequence, such as predicting whether a patient is at risk of a specific disease or determining what product a customer will buy next.

The second approach, which is less common but holds great promise, is known as contrastive learning. Instead of trying to fill in missing pieces, this method teaches the computer to recognize similarity. The system takes two slightly different versions of the same event stream—perhaps by removing a few events or shifting the timing slightly—and learns to recognize that they belong to the same person or entity. It then learns to push apart the representations of different people. This is particularly useful when the goal is to group similar behaviors together, such as identifying clusters of patients with similar health trajectories or finding users with comparable shopping habits. The survey notes that while this method is powerful for creating robust, high-level summaries of behavior, it is currently underutilized in event stream research, partly because designing the right ways to tweak the data without destroying its meaning is a difficult engineering challenge.

The researchers also examined how these methods handle the specific nature of event data, particularly the timing. In many real-world scenarios, events do not happen at regular intervals; a patient might visit the doctor once a year, then three times in a week, while a gamer might log in for hours straight and then disappear for days. Some advanced methods, known as temporal point processes, are designed specifically to model these irregular gaps, treating time itself as a crucial piece of information rather than just a background marker. The survey highlights that while these methods are mathematically rigorous and excellent for predicting exactly when the next event will occur, they sometimes struggle to handle the rich, complex details that come with each event, such as the text of a doctor's note or the image of a product.

Despite the progress made, the paper identifies significant obstacles that prevent these technologies from reaching their full potential. A major issue is the lack of shared resources. In fields like healthcare, data is often locked behind privacy regulations, while in gaming and finance, it is considered a trade secret. This means researchers often have to rely on small, proprietary datasets or repurpose data from unrelated fields, which makes it difficult to compare different methods fairly or to know if a technique will work in a new setting. The authors point out that there is a critical need for open, standardized benchmarks that allow scientists to test their models under the same conditions, much like how athletes compete in the same stadium to see who is truly the fastest.

The survey concludes by outlining a path forward for the field. The researchers suggest that the future lies in developing "domain-agnostic" frameworks—tools that can learn from event streams regardless of whether they come from a hospital, a store, or a video game. They argue that by unifying the different approaches and creating better, more open datasets, the community can move toward building foundational models for event streams. These would be powerful, general-purpose systems capable of learning from vast amounts of unlabeled data and adapting to a wide variety of tasks with minimal human intervention. While the road to such a unified system is long, the survey suggests that the building blocks are already in place, waiting to be connected into a more coherent and powerful whole.

Technical Summary: Towards Unified Approaches in Self-Supervised Event Stream Modeling

Problem Statement
The proliferation of digital interactions across healthcare, e-commerce, finance, and gaming has generated vast volumes of Event Stream (ES) data. Formally defined as continuous, ordered sequences of timestamped events ( $S_u = \{e_{u,i}\}_{i=1}^\infty$ , where $e_{u,i} = (t_{u,i}, d_{u,i})$ ), ES data encapsulates rich contextual information. However, the effective utilization of this data is hindered by two primary challenges: the scarcity of labeled data required for traditional supervised learning, and the fragmented nature of existing research. Current modeling efforts are often siloed within specific domains (e.g., Electronic Health Records vs. user clickstreams), failing to leverage the structural similarities shared by event streams across industries. Furthermore, existing surveys often focus on specific modalities (e.g., time series, tabular data) or specific domains, lacking a unified perspective on Self-Supervised Learning (SSL) tailored specifically for the unique characteristics of ES data, such as irregular sampling and heterogeneous event attributes.

Methodology and Taxonomy
This paper presents a systematic survey and synthesis of SSL methodologies tailored for ES modeling. The authors reviewed approximately 100 works from leading conferences (ICML, NeurIPS, AAAI, KDD) and journals, focusing on studies employing self-supervised pre-training objectives for sequential event data. The core of the paper is a comprehensive taxonomy that organizes these methods into two primary paradigms, visualized in Figure 3:

Predictive SSL: These methods learn representations by predicting or reconstructing masked or missing parts of the event stream.
- Masked Modeling: Inspired by BERT, these models randomly mask events, attributes, or timestamps and train the model to reconstruct them using context. Examples include BEHRT (healthcare) and BERT4Rec (e-commerce).
- Autoregressive (AR) Modeling: These methods predict the next event or attribute based strictly on historical context, aligning with language modeling (GPT) and sequential recommendation systems.
- Temporal Point Processes (TPPs): While originally generative models for event timing, TPPs are included here due to their predictive character. They model the conditional intensity function of event occurrences, offering a principled probabilistic framework for irregular timing (e.g., Neural TPPs, Hawkes processes).
Contrastive SSL: These methods learn a representation space where similar event streams are pulled together and dissimilar ones are pushed apart, often avoiding the reconstruction of noisy low-level details.
- Instance Contrastive: Treats augmented views of the same entity as positive pairs and other entities as negatives. Augmentations include temporal cropping, event dropout, and feature masking.
- Distillation: Uses a teacher-student framework (e.g., BYOL, SimSiam) to align representations of augmented views without explicit negative sampling.
- Feature Decorrelation: Encourages latent dimensions to capture distinct information (e.g., VICReg, Barlow Twins) to reduce redundancy.
- Multimodal Contrastive: Aligns representations across different modalities (e.g., vital signs and clinical notes) associated with the same event stream.

The survey also analyzes the relationship between ES and other modalities (tabular, time series, dynamic graphs), noting that while ES can be transformed into these formats, such transformations often introduce sparsity, lose temporal granularity, or obscure sequential order.

Key Contributions
The paper makes four primary contributions:

Cross-Domain Synthesis: It provides the first (to the authors' knowledge) cross-domain assessment of SSL-based ES modeling, unifying progress across healthcare, gaming, e-commerce, and finance. It argues that ES data possesses unifying structural properties that merit an overarching view, organizing methods by learning objectives rather than application context.
Structured Taxonomy: It presents a detailed taxonomy of SSL paradigms for ES, mapping specific techniques (predictive vs. contrastive) to the downstream tasks they best serve (e.g., autoregressive for next-event prediction, masked modeling for classification, contrastive for entity-level clustering).
Resource and Benchmark Guide: The authors compile an overview of widely used public datasets (e.g., MIMIC, Amazon, StackOverflow) and identify critical limitations in current benchmarking practices, such as the lack of multi-domain benchmarks and standardized evaluation protocols.
Future Research Roadmap: The paper outlines critical open issues, including the need for domain-agnostic frameworks, better handling of timestamps, and the development of robust augmentations.

Results and Findings
The survey reveals that predictive methods currently dominate the ES landscape, particularly masked modeling and autoregressive approaches, largely due to their natural alignment with the sequential structure of event data and the ease of transferring NLP architectures (Transformers) to event sequences. In contrast, contrastive methods remain underutilized, despite their potential to produce robust, entity-level embeddings that are less sensitive to noise in individual event attributes.

Key observations regarding domain applicability include:

Healthcare: Heavily utilizes masked modeling (e.g., BEHRT) and is increasingly exploring multimodal contrastive learning to integrate structured vitals with unstructured notes.
E-commerce: Dominated by autoregressive and masked modeling for sequential recommendation (e.g., BERT4Rec, CL4SRec).
Gaming and Finance: These domains are underrepresented in public literature due to proprietary data constraints. Gaming research often relies on AR objectives for action prediction, while finance focuses on masked modeling for transaction analysis (e.g., BERT4Eth).

The paper notes that while controlled comparisons of SSL paradigms under matched conditions are lacking, observed patterns suggest that task alignment is crucial: autoregressive objectives suit next-event prediction, while masked modeling benefits tasks requiring holistic sequence understanding.

Significance and Claims
The paper claims that by unifying disparate research efforts and highlighting cross-domain synergies, this survey aims to accelerate innovation and improve reproducibility in the ES modeling community. It posits that the field is currently fragmented, with research efforts often replicating work across verticals without leveraging shared structural similarities.

The authors emphasize that their guidance on paradigm selection is based on observed patterns in the literature rather than systematic empirical evidence, as controlled comparisons are currently scarce. They call for the development of scalable, domain-agnostic SSL frameworks and standardized benchmarks (such as the emerging EBES framework) to move the field from heuristic recommendations to empirically grounded best practices. Ultimately, the paper argues that advancing unified SSL approaches is essential for unlocking the full potential of ES data in real-world applications, enabling models to learn from raw event logs with minimal human annotation.

More like this