🤖 machine learning

Queryable Self-Organizing Maps: A Database Abstraction for Topology-Driven Data Exploration

This paper introduces "queryable data maps" and presents MapDB, a prototype system that integrates Self-Organizing Maps directly into database management systems to enable interactive, topology-driven data exploration using SQL without leaving the DBMS.

Original authors: Denis Mayr Lima Martins, Gottfried Vossen

Published 2026-07-28✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Denis Mayr Lima Martins, Gottfried Vossen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, chaotic library where every book is a single piece of information about a person, a product, or an event. In this library, the books aren't organized by title or author; they are scattered randomly across millions of shelves. If you wanted to find all the books about "people who love hiking and live in rainy cities," you'd have to pull out every single book, read it, and check. That's what it's like for computers trying to make sense of huge, messy databases with hundreds of different facts about each item. This is the world of "high-dimensional data." To help humans navigate this chaos, scientists have long used a clever trick called a Self-Organizing Map (SOM). Think of a SOM as a magical, living floor plan. It takes all those scattered books and automatically arranges them on a flat, two-dimensional grid. Similar books end up next to each other, forming cozy neighborhoods. If you look at this map, you can instantly see where the "hiking lovers" live (a dense cluster), where the "rainy city dwellers" hang out, and where the lonely, weird books are (sparse regions). It turns a confusing pile of data into a clear picture.

However, there's a catch. Usually, this magical map is drawn by a separate computer program (like a Python script) sitting outside the main library. Once the map is drawn, the library staff (the database) doesn't know it exists. You can't ask the library, "Show me all the books near the hiking neighborhood," because the library doesn't speak the map's language. You have to leave the library, look at the map, figure out what you want, and then go back to write a new, complicated list of rules to find those books. It's like having a treasure map that you can't use to navigate the actual island. This paper, titled "Queryable Self-Organizing Maps," asks a simple but powerful question: What if we could build the map inside the library itself? What if the map became a permanent part of the database, so you could ask it questions just like you ask for a specific book?

The authors, Denis Mayr Lima Martins and Gottfried Vossen, introduce a new system called MapDB to solve this problem. They propose treating these learned maps not as temporary drawings, but as "intensional data"—meaning they are persistent, living objects stored right alongside the data they describe. In MapDB, the map isn't just a picture; it's a set of tables that the database understands. This allows users to use standard database commands (SQL) to explore the map. You can ask the database to "find the dense neighborhoods," "show me the boundaries between different groups," or "zoom in on a specific spot and tell me which original books belong there."

The paper demonstrates that this idea works. The researchers built a prototype using a database engine called DuckDB and tested it with different types of data, including a dataset about adult census records and a complex set of sales data (TPC-H). They found that training these maps directly inside the database is possible and reasonably fast for moderate-sized datasets. For example, on their test data, the system could train a map in about 20 seconds. They also discovered that if you save (or "materialize") certain parts of the map, like the boundaries between groups, the database can answer questions about them much faster—up to 7.5 times faster in some cases—because it doesn't have to recalculate the math every time.

Crucially, the paper shows that this approach doesn't just speed things up; it changes how we explore data. Instead of guessing which filters to apply, a user can look at the map, see a strange, empty spot, and ask the database, "What is in this empty area?" or "Who are the people living right next to this crowded neighborhood?" The system then instantly returns the actual records that match that location. The authors suggest that this turns data exploration from a game of "guess and check" into a guided tour. While the paper notes that for extremely massive datasets, the system might need future optimizations to stay fast, the experiments confirm that keeping the map inside the database is a viable and powerful way to understand complex data. It suggests that the future of data analysis might not be about building better separate tools, but about making the database itself smart enough to organize and explain its own contents.

Technical Summary: Queryable Self-Organizing Maps

Problem Statement

In modern data systems, Self-Organizing Maps (SOMs) serve as powerful exploratory tools for high-dimensional data, organizing objects into a two-dimensional topology that reveals clusters, gradients, sparse/dense regions, and boundaries. However, a significant gap exists in current workflows: SOMs are typically trained and visualized outside the Database Management System (DBMS) using transient artifacts (e.g., Python or R scripts). Consequently, the learned topological structure is disconnected from the relational data it summarizes. This separation prevents natural joins between map regions and source tuples, lacks DBMS governance for map versions, and hinders the direct exploitation of learned topology for exploratory recommendations. Analysts often struggle to define precise SQL predicates without first understanding the "shape" of high-dimensional data, a task for which conventional SQL is ill-suited.

Methodology

The paper introduces MapDB, a lightweight prototype that treats SOMs as intensional data—persistent, learned topological artifacts stored and queried directly within the relational database.

Core Abstraction: The Queryable Data Map

The authors define a queryable data map $M$ as a tuple $(V, E, W, A, S)$ , where:

$V$ : Set of map units (neurons).
$E$ : Neighborhood relation (topology edges).
$W$ : Representative parameters (prototype vectors).
$A$ : Object-to-neuron assignments (Best Matching Unit).
$S$ : Derived summaries (e.g., hit counts, U-Matrix values, component planes).

Each component is represented as a relational table, making the entire map structure queryable via standard SQL operations.

System Architecture

MapDB separates the system into four layers:

Map-Aware SQL Syntax: A declarative interface for map creation, training, and analysis (e.g., CREATE MAP, TRAIN MAP, map_dense_regions).
Logical Map Operations: High-level operations such as BMU assignment, neighborhood traversal, and U-Matrix inspection.
Persistent Map State: The physical storage of SOM neurons, weights, edges, and assignments in core relations (MAP_NEURON, MAP_WEIGHT, MAP_EDGE, MAP_ASSIGNMENT, MAP_SUMMARY).
Relational Execution: The backend DBMS (implemented in DuckDB) executes these operations using standard relational algebra (joins, aggregations, window functions).

Training and Querying

Training: MapDB utilizes batch SOM training. The Best Matching Unit (BMU) for an input object is computed via relational joins and window ranking to find the minimum Euclidean distance to prototype vectors. Prototype updates are performed as neighborhood-weighted averages, implemented via joins over topology edges and input features.
SQL Extensions: The system provides specific extensions for:
- Density Analysis: Identifying dense neurons and regions.
- Boundary Detection: Querying U-Matrix values to find sharp transitions between prototypes.
- Component Planes: Visualizing how individual features vary across the map as persistent views.
- Region Drill-Down: Traversing the topology to retrieve source tuples mapped to specific neurons or neighborhoods.

Key Contributions

The paper makes four primary contributions:

Intensional Data Concept: It argues that persistent topology can provide a queryable organization over relational tuples, treating learned maps as first-class database objects rather than external artifacts.
MapDB Prototype: It presents a relational representation and SQL-resident execution substrate for SOM neurons, prototypes, topology, assignments, and summaries.
Map-Aware SQL Extensions: It introduces a compact family of SQL extensions for map creation, density analysis, U-Matrix inspection, component-plane queries, neighborhood traversal, region drill-down, and selective summary materialization.
Comprehensive Evaluation: It provides an evaluation covering scalability, comparisons with flat-clustering baselines (KMeans), materialization trade-offs, map-size sensitivity, TPC-H scaling, and case studies on single-table and multi-table settings.

Experimental Results

The evaluation was conducted using DuckDB with Python orchestration, testing on Synthetic data, the Adult Census dataset, and TPC-H benchmarks.

Scalability: Relational SOM processing is feasible for moderate analytical workloads. At the largest tested configuration, MapDB required approximately 20 seconds for training. The dominant cost is the BMU assignment step.
Quality vs. Flat Clustering: MapDB achieves competitive results in quantization error and silhouette scores compared to Python Batch SOM, KMeans, and MiniBatch KMeans. Crucially, unlike flat clustering, MapDB preserves explicit adjacency, enabling neighborhood expansion and topological traversal.
Materialization Trade-offs: Querying materialized artifacts (e.g., pre-computed U-Matrices) yields significant speedups. U-Matrix queries showed the highest median speedup (7.57x) because materialization avoids the complex joins and distance aggregations required for on-the-fly computation.
Case Studies:
- Adult Dataset: The SOM revealed non-uniform density and strong topological boundaries. Component planes showed systematic gradients (e.g., age varying from left to right), allowing analysts to drill down from specific regions to source records with distinct demographic profiles.
- TPC-H (Multi-Table): The system successfully mapped customer-level feature views derived from joins (Customer, Orders, Lineitem, etc.). The resulting map organized customers by purchasing intensity (orders, total spend) and identified localized structures for logistics and pricing behaviors.
Region Drill-Down: The system successfully joined map regions back to source data, identifying dominant labels and feature contrasts (e.g., dense regions in TPC-H corresponding to high-activity "BUILDING" market segments).

Significance and Claims

The paper claims that treating SOMs as intensional data bridges the gap between relational querying and exploratory understanding. By keeping the learned topology inside the DBMS, MapDB enables:

Direct Integration: Map regions can be naturally joined with source tuples.
Governance: Map versions are managed by the DBMS.
Exploratory SQL: Users can formulate queries based on topological properties (density, boundaries, neighborhoods) without leaving the database environment.

The authors position MapDB not as a performance-optimized replacement for specialized SOM libraries, but as a demonstration of a database abstraction. The significance lies in shifting the paradigm where learned structures are no longer opaque external objects but persistent, queryable components of the database itself. This opens new avenues for maintenance, physical design, and optimization of learned analytical structures, including vector quantization and graph embeddings.