Technical Summary: SemPiper: Interactive Code Synthesis for Semantic Operators in Machine Learning Pipelines

Problem Statement

Machine learning (ML) pipeline development is characterized by extensive, repetitive, and error-prone data preparation, feature engineering, and integration across heterogeneous sources (e.g., data lakes, warehouses, log files). While Large Language Models (LLMs) have shown promise in code synthesis, current chat-based interfaces offer limited control over pipeline behavior. They often produce code that is difficult to optimize, integrate into production systems, or verify for safety. Furthermore, relying on LLMs for every inference step in a pipeline is computationally expensive and introduces latency. The core challenge is to integrate the flexibility of LLMs for high-level data-centric tasks while maintaining the determinism, optimizability, and efficiency required for standard ML pipelines.

Methodology: The SemPipes Framework

The paper introduces SemPipes, a programming model that extends standard tabular ML pipelines with Semantic Data Operators (SemOps). These are declarative, LLM-powered operators that allow developers to specify high-level natural language instructions for data-centric operations, while the system synthesizes concrete Python implementations.

Core Architecture

1. Declarative Interface: SemPipes pipelines are written in Python using standard data science libraries (pandas, NumPy, scikit-learn, skrub). Developers selectively delegate specific operations to SemOps by providing natural language instructions (e.g., "Normalise to the ISO-3166 standard").
2. Training-Time Synthesis: Unlike systems that rely on LLMs during inference, SemPipes synthesizes specialized operator implementations only at pipeline training time.
   • The synthesis is conditioned on three factors: dataset characteristics, user natural language instructions, and the pipeline context (downstream task and model type).
   • The system generates specific Python code (e.g., pandas operations, calls to pre-trained models like HuggingFace transformers) based on these conditions.
   • This approach limits LLM usage to a number of calls proportional to the number of semantic operators, eliminating LLM dependency during inference.
3. Code Generation Strategies:
   • For feature generation, SemPipes produces CPU-based pandas code.
   • For multimodal extraction, SemOps leverage pre-trained models on local GPUs (e.g., zero-shot classification or visual question answering).
4. Validation and Safety: Synthesized code undergoes operator-specific validation, including syntax checking and execution on data samples to verify output constraints (e.g., dataframe shape).

Optimization via Evolutionary Search

SemPipes automates the iterative refinement of data operations. Once a pipeline is defined, the system employs an evolutionary search procedure (e.g., Monte Carlo Tree Search) to optimize the synthesized operator code:

• Mutation: The system generates candidate implementations by mutating the current code using reflective prompting, guided by prior performance scores and a tree-structured memory.
• Fitness Function: The search evaluates candidates based on their impact on the downstream model's performance on a validation set.
• Selection: Improved variants are selected to refine the pipeline, effectively automating the manual process of rewriting feature engineering code.

Key Contributions and Demonstration (SemPiper)

The paper presents SemPiper, an interactive demonstration interface that visualizes the SemPipes workflow. The system is demonstrated across three end-to-end scenarios:

1. E-commerce Fraud Detection:

   • Task: Binary classification of fraudulent shopping baskets using multi-table retail data.
   • SemOps: sem_fillna (imputing missing manufacturers via inference), sem_extract_features (deriving brand risk scores from text), sem_gen_features (synthesizing new features), and sem_agg_features (aggregating product data to basket level).
   • Synthesis: The system selected a non-parametric KNNImputer for missing values and utilized zero-shot classification models for brand risk scoring.

2. Multimodal Museum Catalog Analysis:

   • Task: Predicting the cultural origin of artworks using tabular metadata and unstructured text descriptions.
   • SemOps: sem_extract_features (converting heterogeneous date strings to structured temporal representations), sem_gen_features (enriching data with text length and temporal features), and sem_refine (standardizing noisy object names).
   • Synthesis: Combined HuggingFace models with rule-based parsing and regular expressions to extract structured JSON fields from raw text.

3. Real Estate Price Prediction:

   • Task: Regression to predict house prices in California using tabular data and house images.
   • SemOps: sem_clean (detecting and correcting anomalies in square footage), sem_extract_features (deriving exterior features from images), and sem_gen_features (synthesizing property and location-level attributes).
   • Synthesis: Used visual question answering models (e.g., Salesforce/blip-vqa-base) to extract semantic image features and applied IQR-based capping for outlier removal.

Results and Observations

The demonstration highlights the system's ability to:

• Synthesize Context-Aware Code: Generate specific implementations (e.g., selecting specific pre-trained models or statistical methods) based on the input data and instructions.
• Visualize Optimization Trajectories: The interface displays the evolutionary search tree, allowing users to trace how candidate implementations evolve, their validation performance, and the specific code mutations that led to improvements.
• Improve Performance: In the house price prediction scenario, the optimizer successfully refined feature extraction, moving from coarse features (e.g., has_visible_garage) to semantically rich features (e.g., architectural_style, landscape_quality), which correlated with better downstream model performance.

Significance and Claims

The paper claims that SemPipes offers a controllable, optimizable, and practical integration of LLMs into ML pipeline development. Its significance lies in:

• Bridging the Gap: It moves beyond coarse chat-based coding to a declarative model where LLMs act as specialized, context-aware data operators within a standard Python ecosystem.
• Efficiency: By synthesizing code at training time rather than relying on LLMs during inference, it avoids the latency and cost associated with continuous LLM invocation.
• Iterative Development: It enables semi-automated pipeline construction where developers can define high-level intent, while the system handles the concrete implementation and subsequent optimization via evolutionary search.
• Transparency: Unlike "black box" LLM generation, SemPipes exposes the generated code, computational graphs, and optimization paths, allowing developers to inspect, modify, and understand the data transformations.

The authors position SemPipes as a step toward making data preparation less tedious and more engineering-friendly, leveraging LLMs for the "what" (intent) while retaining human and algorithmic control over the "how" (execution and optimization).