optimade-maker: Automated generation of interoperable materials APIs from static data

The paper introduces **optimade-maker**, a lightweight toolkit that automates the conversion of diverse raw atomistic datasets into interoperable, OPTIMADE-compliant REST APIs, thereby lowering technical barriers and enabling scalable, FAIR data integration across community-contributed and curated materials databases.

The Gist

Imagine the world of materials science as a massive, bustling library. In this library, scientists are constantly discovering new "books" (materials like new alloys, batteries, or solar cells) and writing down their properties.

The Problem: A Library of Mismatched Catalogs
The trouble is, every librarian (database provider) has their own unique way of organizing and describing these books.

• One librarian uses a card catalog with handwritten notes.
• Another uses a digital app that only speaks French.
• A third keeps their books in a locked room with a custom key.

If you want to find a specific type of book, you have to learn how to talk to every single librarian individually. It's exhausting, slow, and confusing. This is the current state of materials data: thousands of databases, but no common language to search them all at once.

The Solution: The "Universal Translator" (OPTIMADE)
A few years ago, the scientific community invented a "Universal Translator" called OPTIMADE. It's a standard set of rules that says, "If you want to be part of this global library, you must label your books and organize your shelves exactly like this."

This allows a single search tool to ask, "Show me all books about batteries," and instantly get answers from 20 different libraries simultaneously.

The New Tool: The "Instant Library Builder" (optimade-maker)
Here is the catch: While the Universal Translator (OPTIMADE) is great, becoming a compliant library is hard work. It requires hiring IT experts, buying special software, and spending months reorganizing your shelves. Many small research groups or individual scientists have amazing data, but they can't afford the time or money to build a compliant library. Their data stays locked in "raw files" (like a messy pile of papers in a drawer).

Enter optimade-maker.

Think of optimade-maker as a magic, automated librarian robot.

1. It Takes the Mess: You give the robot a box of raw data (your messy pile of papers, simulation results, or crystal structure files).
2. It Reads the Instructions: You give it a simple "recipe card" (a configuration file) that says, "This file is the title, this column is the price, and this folder holds the pictures."
3. It Builds the Library: The robot instantly sorts everything, labels it according to the Universal Translator rules, and sets up a digital storefront (an API) where anyone can search your data immediately.

Real-World Examples of the Robot in Action

• The "Instant Upload" Feature: Imagine a researcher uploads a new dataset to the "Materials Cloud" (a public digital storage). With optimade-maker, the moment they hit "upload," the robot automatically builds a search engine for that specific dataset. Suddenly, that data is searchable by the whole world, without the researcher writing a single line of complex code.
• The "Big Database" Upgrade: The paper also shows how this robot helped upgrade two massive, commercial libraries (the Cambridge Structural Database and the Inorganic Crystal Structure Database). These libraries have millions of entries. Instead of manually re-cataloging them, the robot created a bridge, translating their complex internal formats into the Universal Translator language, allowing researchers to search them alongside thousands of other free databases.

Why This Matters
Before this tool, sharing data was like trying to send a fax to a smartphone; it just didn't work well. optimade-maker turns that fax machine into a smartphone instantly.

It lowers the barrier to entry so that:

• Small researchers can share their work easily.
• Big databases can connect to the global network.
• AI and discovery tools can scan everything at once to find the next breakthrough material.

In short, optimade-maker is the tool that turns a messy pile of raw data into a perfectly organized, globally searchable library in seconds, making the future of materials discovery faster and more open for everyone.

Technical Summary

1. Problem Statement

Atomistic structural data is fundamental to materials science, yet it is increasingly fragmented across diverse repositories (e.g., COD, ICSD, CSD, Materials Cloud) and file formats. While Application Programming Interfaces (APIs) exist for some databases, they are often custom-built and lack interoperability. This forces client applications to implement support for multiple incompatible standards, hindering cross-database analysis and data-driven discovery.

Although the OPTIMADE (Open Databases Integration for Materials Design) consortium has established a standard REST API specification to unify access, deploying and maintaining a compliant API remains technically demanding. It typically requires dedicated infrastructure, database management systems, and significant engineering effort. This creates a high barrier to entry for individual researchers, small groups, or static datasets (e.g., simulation outputs) that wish to share data in a FAIR (Findable, Accessible, Interoperable, Reusable) manner.

2. Methodology

The authors present optimade-maker, a lightweight Python toolkit designed to automate the generation of OPTIMADE-compliant APIs directly from raw atomistic structure and property data files.

Core Components

The toolkit is built on top of optimade-python-tools and consists of four main modules:

1. Configuration (optimade_maker.config): Uses a simple YAML file (optimade.yaml) to define the data schema. It specifies file paths, entry types (e.g., structures), and property definitions (including data types, units, and descriptions).
2. Conversion (optimade_maker.convert): A pipeline that ingests raw data (archives like ZIP/TAR), parses structure files using ASE (Atomic Simulation Environment) or Pymatgen, and maps property data from CSV/JSON files. It generates a standardized OPTIMADE JSON Lines file (a new format introduced in OPTIMADE v1.3.0).
3. Parsing (optimade_maker.parsers): A registry of parsers that attempt to read various file formats (CIF, XYZ, XSF, etc.) and map them to intermediate objects (ASE Atoms or Pymatgen Structures), which are then converted to OPTIMADE entries.
4. Serving (optimade_maker.serve): Launches an OPTIMADE API server. By default, it uses an in-memory MongoDB (via MongoMock) for rapid local deployment, but it can be configured to use external databases for production environments.

Key Workflow Features

• Identifier Generation: Automatically generates unique structure IDs based on file paths relative to the configuration file using a deterministic rule (removing common prefixes/suffixes).
• AiiDA Integration: Can directly ingest data from AiiDA (a workflow management infrastructure) by querying specific groups and extracting properties (e.g., band gaps) from provenance graphs using AiiDA's query builder.
• CLI Interface: Provides two primary commands:
  • optimake convert: Transforms raw data into the standard JSON Lines format.
  • optimake serve: Starts the API server immediately.

3. Key Contributions

1. Automated API Generation: The primary contribution is a toolkit that eliminates the need for manual database schema design and API coding, allowing researchers to expose raw datasets as OPTIMADE APIs with minimal configuration.
2. Materials Cloud Archive Integration: The authors implemented an automated service on the Materials Cloud Archive. When researchers upload a dataset with a valid optimade.yaml, the system automatically:
   • Processes the data.
   • Deploys a dedicated OPTIMADE API endpoint.
   • Links the dataset to the Materials Cloud OPTIMADE Client for interactive browsing.
3. Large-Scale Database Transformation: The toolkit was used to create data transformation pipelines for two major commercial databases:
   • Cambridge Structural Database (CSD): Mapping organic/metal-organic structures and chemical identifiers.
   • Inorganic Crystal Structure Database (ICSD): Mapping inorganic structures and diffraction metadata.
   • These pipelines enable unified, cross-searchable access for UK academics via the Physical Sciences Data Infrastructure (PSDI).
4. New Client Interface: A complete rewrite of the Materials Cloud OPTIMADE Client (from Python to JavaScript) was released, offering improved performance, interactive periodic table filtering, and 3D visualization.

4. Results and Use Cases

• Materials Discovery Projects: The toolkit successfully served datasets from high-throughput screening projects, such as the discovery of ternary electride materials ($A_2BC_2$) and optical response materials. These datasets, containing over 14,000 compounds and associated DFT properties (e.g., ELF, formation energy), became immediately queryable via standard OPTIMADE filters.
• Cross-Provider Queries: The system enables complex queries across different providers. For example, users can query for structures with specific convex hull distances and electron localization function values across multiple datasets simultaneously.
• Scalability: The system handled the transformation of millions of entries for the CSD and ICSD, demonstrating its ability to manage large-scale, continuously updated databases through periodic snapshot updates.

5. Significance

• Lowering Barriers to FAIR Data: optimade-maker significantly reduces the technical overhead required to publish materials data. It allows researchers to make static datasets interoperable without building complex backend infrastructure.
• Ecosystem Growth: By automating the creation of compliant APIs, the toolkit accelerates the growth of the OPTIMADE ecosystem, currently indexing over 25 million materials across 20 providers.
• Interoperability: It bridges the gap between static file-based data (common in academic research) and dynamic, queryable API ecosystems, enabling seamless integration with machine learning pipelines and discovery tools.
• Future-Proofing: The modular design allows for easy integration of new entry types (e.g., trajectories, files) and community-defined properties (e.g., SMILES annotations), ensuring the framework evolves with the OPTIMADE specification.

In conclusion, optimade-maker represents a critical step toward a fully scalable, FAIR materials data ecosystem by democratizing the ability to publish interoperable data, thereby accelerating materials discovery and design.