💻 computer science

Schema-Agnostic Process Trace Construction: From Raw Tables to Execution Behavior

This paper proposes a schema-agnostic pipeline that automatically reconstructs high-fidelity process execution traces from raw, loosely connected relational tables by statistically identifying key and temporal attributes, discovering inter-table connections, and utilizing a Temporal Convolutional Network to model event ordering, thereby eliminating the need for predefined schemas or domain templates in dynamic information systems.

Original authors: Joel Lim Zhi Quan, Tan Kar Way, Lau Hoong Chuin

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joel Lim Zhi Quan, Tan Kar Way, Lau Hoong Chuin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to reconstruct the story of a bank robbery. In a perfect world, the police would hand you a neat, chronological diary where every entry says exactly who did what and when.

But in the real world, the evidence is scattered. You have a pile of loose receipts, a stack of security camera logs, a few handwritten notes, and a spreadsheet of phone calls. None of these documents talk to each other. The receipts don't have names, the camera logs don't have timestamps, and the phone notes are written in a different language. Furthermore, the bank keeps changing its filing system every week.

This is the problem the authors of this paper are solving. They are dealing with modern computer systems (like banks or large companies) where data is messy, scattered across many different tables, and constantly changing. Traditional methods for analyzing these systems fail because they demand a perfect, pre-organized map (a "schema") that simply doesn't exist.

Here is how their solution works, broken down into simple steps:

The Problem: The "Messy Filing Cabinet"

In old-school computer systems, data was like a well-organized library. Every book had a clear label, and you knew exactly which shelf it belonged to.
In modern systems, data is like a hoarder's attic.

No Labels: You can't easily tell which piece of data belongs to which customer (missing "keys").
Scattered Clues: The story of a single transaction is split across five different tables.
Confusing Timelines: One table says "10:00 AM," another says "10:05 AM," and a third just says "Yesterday."
Constant Renovation: The attic is being rearranged while you are trying to clean it.

Because of this, building a clear timeline of what happened (called a "process trace") usually requires a human expert to manually stitch it all together, which is slow, expensive, and prone to errors.

The Solution: A "Schema-Agnostic" Detective

The authors built an automated pipeline that acts like a super-smart detective who doesn't need a map. Instead of asking, "Where is the map?" they just look at the evidence itself to figure out the story.

Here are the four steps their "detective" takes:

1. Spotting the Clues (Profiling)

First, the system scans every column of data to guess what it is.

The ID Hunt: It looks for columns that look like unique names (like a customer ID). It checks: "Is this value unique? Is it always there? Does it look like a name?"
The Time Hunt: It looks for columns that look like dates. It checks: "Does this look like a timestamp? Is it consistent?"
Analogy: Imagine sorting a pile of mixed-up puzzle pieces. The detective doesn't need the picture on the box; they just look at the shape of the pieces to guess which ones are the sky and which are the grass.

2. Connecting the Dots (Relationship Discovery)

Since there are no official "connect the dots" lines (foreign keys), the system uses statistical signals.

It compares columns from different tables. If Table A has a list of numbers and Table B has a list of numbers that match perfectly, the system assumes they are connected.
It ignores the "official" rules and looks at the actual data patterns.
Analogy: If you find a receipt in one pocket and a matching credit card statement in another, you know they belong to the same person, even if they aren't stapled together.

3. Building the Timeline (Sequencing)

Once the system knows which tables are connected, it gathers all the events for a single "case" (like a specific customer order).

It sorts these events by time.
If the times are confusing or missing, it uses logic to guess the order.
Analogy: The detective takes all the scattered notes, receipts, and logs for one specific robbery and arranges them on a table to see the sequence of events.

4. Learning the Pattern (The "Brain" - TCN)

This is the most advanced part. Sometimes, the timestamps are too messy to tell which event happened first.

The system uses a special type of AI called a Temporal Convolutional Network (TCN). Think of this as a pattern-recognition engine.
It looks at thousands of past examples to learn: "Usually, when Event A happens, Event B follows."
Even if the clock is broken, the AI can predict the next step based on the flow of the story.
Analogy: If you see someone putting on a coat, grabbing keys, and opening a door, you know they are about to leave, even if you didn't see the exact moment they walked out. The AI learns these "story flows."

The Results: How Good is the Detective?

The authors tested this system on fake data (simulating messy banks), standard benchmarks, and a real industry dataset.

Accuracy: It correctly predicted the next step in a process 85% of the time.
Recovery: It managed to find and reconstruct about 82% of the correct order of events, even when the data was missing or messy.
Resilience: When the data got "drifted" (changed names, missing dates), the system kept working, while traditional methods fell apart.

Why This Matters

The paper argues that we need to stop waiting for perfect data before we can analyze it. Instead of forcing messy real-world data into a rigid, pre-defined box, we should let the data speak for itself.

By removing the need for a perfect "schema" (the map), this approach allows companies to automatically understand their own systems, even if those systems are messy, constantly changing, or poorly documented. It turns a chaotic pile of evidence into a clear, readable story without needing a human to do all the heavy lifting.

Technical Summary: Schema-Agnostic Process Trace Construction

Problem Statement

Modern Information Systems (IS), particularly in OLTP environments, are characterized by rapidly evolving, loosely coupled relational tables. In these settings, traditional assumptions of stable schemas, explicit primary/foreign keys, and curated event logs often fail. Schemas drift, surrogate keys are inconsistently applied or refactored, and execution traces are dispersed across multiple tables with sparse keys and ambiguous or multiple timestamp fields.

Consequently, reconstructing process execution traces (event logs) for tasks like auditing, compliance, and process mining requires costly, fragile manual intervention involving ad-hoc joins and timestamp rules. Existing approaches typically rely on predefined schemas, reliable PK/FK structures, or domain-specific templates, rendering them inapplicable to key-sparse, heterogeneous, and continuously changing environments. The core problem addressed is how to automatically construct high-fidelity execution sequences (cases, events, and temporal ordering) directly from raw relational data without relying on predefined schemas, reliable foreign keys, or domain expertise.

Methodology

The authors propose a four-stage, schema-agnostic pipeline that infers inter-table links from statistical signals, assembles temporal fields, and learns ordering patterns. The pipeline operates without requiring full schema reconstruction or ontological mappings.

Stage 1: Profiling Identifier and Timestamp Columns

The pipeline profiles columns in each table to estimate their likelihood of functioning as case identifiers or timestamps.

Identifier Scoring: A score $spk(c)$ is computed based on distinctness ( $u$ ), completeness ( $\gamma$ ), and string-length stability ( $\lambda$ ). Columns must score highly on all three metrics to be retained as candidate identifiers.
Timestamp Scoring: A score $sdt(c)$ is calculated using a weighted sum of percentile ranks for dispersion and variation metrics (completeness, distinctness, standard deviation, inter-quartile range, and entropy).
Output: The top- $k$ candidate identifiers and timestamp columns are selected for each table.

Stage 2: Relationship Discovery (Column–Column Links)

Instead of relying on declared foreign keys, the pipeline discovers inter-table connections using type-aware statistical similarity.

Scoring Mechanism: For every cross-table column pair $(c_i, c_k)$ $(c_{i}, c_{k})$ , a link score $s(c_i, c_k)$ $s (c_{i}, c_{k})$ is computed as the product of four factors:
1. Type Compatibility (TC): Downweights pairs with weakly compatible data types.
2. Column Quality (Q): Penalizes pairs involving low-completeness or low-distinctness columns.
3. Coverage (U): Ensures overlap is not driven by rare fringe values.
4. Value Similarity (R): Measures actual value alignment using Jaccard Similarity (JS) for symbolic data and a JS/Kolmogorov-Smirnov (KS) blend for numeric distributions.
Link Formation: Pairs exceeding a threshold $\tau_S$ are retained to form an undirected multigraph of candidate links.

Stage 3: Temporalization and Sequencing

Using the discovered link set, the pipeline propagates sampled case identifiers across reachable tables to collect corresponding rows and timestamp candidates.

Event Assembly: Each (row, timestamp) pair is treated as a provisional event.
Ordering: Events for a specific case are sorted by timestamp. Ties are broken deterministically using table names and row identifiers. Adjacent duplicate labels are collapsed, while non-adjacent repetitions are preserved.
Context: Optional business attributes (e.g., region, product) are embedded into context vectors to condition subsequent learning without altering the event order.
Output: Provenance-rich event traces in JSON format, exportable to XES or liftable to OCEL.

Stage 4: Sequence Learning (Ordering Across Tables)

To address cases where timestamps are ambiguous, missing, or insufficient to determine order, the pipeline employs a Causal Temporal Convolutional Network (TCN).

Learning Objective: The TCN predicts the next event $e_{t+1}$ given a window of past events and an optional context vector.
Precedence Extraction: By aggregating next-event predictions, a transition matrix is formed. Directed edges $i \to j$ are added when the probability of $j$ following $i$ significantly exceeds the reverse. Pairs with similar bidirectional frequencies are marked as exhibiting Bidirectional Order Variability (BOV).
Output: A minimal precedence graph summarizing flow relations, which can be consumed by standard process mining tools.

Key Contributions

Schema-Agnostic Pipeline: A novel end-to-end approach that reconstructs process traces from raw, key-sparse relational data without requiring predefined schemas, ER diagrams, or domain templates.
Statistical Link Discovery: A method for inferring table relationships using type-aware statistical similarity (JS/KS) rather than relying on explicit foreign keys, making it robust to schema drift and missing keys.
Learned Precedence: The application of TCNs to learn cross-table ordering and flow relations directly from reconstructed traces, handling scenarios where timestamps are insufficient or conflicting.
Operational Robustness: A pathway to generate event logs that remain robust to missing timestamps, homonyms, and partial keys, enabling automated log construction in dynamic IS environments.

Results

The pipeline was evaluated on TPC-H/E benchmarks, synthetic corpora (Synth-Light, Synth-Heavy), and a real industry dataset (Industry-1).

Event Recovery (RQ1): The pipeline achieved high event-set recovery, with Position-Independent Token Accuracy (PITA) ranging from 0.82 (Synth-Heavy) to 1.00 (TPC-H). It correctly predicted the next event with 85% accuracy.
Precedence Learning (RQ2): The TCN-based approach outperformed n-gram and LSTM baselines in next-event prediction and conformance metrics. It recovered approximately 82% of ground-truth precedence relations (F1 score 0.74–0.82) and demonstrated superior ability to capture cross-table flow relations compared to rule-based or schema-centric approaches.
Change Resilience (RQ3): Under controlled drift (attribute renaming and timestamp missingness), the full pipeline retained over 80% of its clean-data performance metrics, whereas a deterministic baseline dropped below 50%. This demonstrates significant resilience to schema and data evolution.
Stage Attribution: Ablation studies confirmed that Stages S1–S3 drive reconstruction quality, while Stage S4 (TCN) significantly improves the fidelity of precedence relations, particularly in capturing long-range dependencies and context-specific flows.

Significance and Claims

The paper claims to offer a generalizable and scalable pathway for automated reconstruction of execution behavior in dynamic IS environments. It challenges the long-standing assumption in IS engineering that reliable event logs presuppose clean schemas and curated keys. By shifting from a "schema-first" to a "data-first" approach, the authors argue that it is possible to construct high-quality logs even as schemas drift and systems evolve.

The work suggests a new IS engineering paradigm where operational schemas are allowed to evolve freely, while a data-driven log-engineering layer preserves continuity for analysis, auditing, and redesign. The authors position this as a solution to the bottleneck of manual trace construction in modern, loosely coupled OLTP systems, enabling process mining and compliance monitoring in environments previously considered too noisy or unstructured for such analysis.