💻 computer science

A Domain-Specific Language for LLM-Driven Trigger Generation in Multimodal Data Collection

This paper proposes a declarative framework that leverages large language models to translate natural language user requests into verifiable domain-specific language programs, enabling efficient, intent-driven, and selective on-device collection of multimodal sensor data while reducing storage costs and execution latency compared to passive logging.

Original authors: Philipp Reis, Philipp Rigoll, Martin Zehetner, Jacqueline Henle, Stefan Otten, Eric Sax

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Reis, Philipp Rigoll, Martin Zehetner, Jacqueline Henle, Stefan Otten, Eric Sax

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car equipped with a super-smart camera and a bunch of other sensors (like LiDAR and radar). This car is constantly recording everything it sees: every pedestrian, every traffic light, every pothole, and even the color of the sky.

The Problem: The "Firehose" Effect
Right now, most of these cars record everything all the time. It's like trying to drink from a firehose.

Too much data: You end up with terabytes of useless footage (like recording the empty road for 10 hours straight).
Too expensive: Storing all this data costs a fortune.
Too slow: If you want to find a specific moment (like "a dog running across the street in the rain"), you have to sift through mountains of boring video later.

The Solution: A Smart "Tripwire" System
This paper proposes a new way to collect data. Instead of recording everything, the car should only record when something specific happens. But here's the catch: telling a computer exactly what to look for usually requires a highly skilled programmer to write complex code.

The authors created a system that lets you ask for what you want in plain English, and the computer figures out the rest.

The Three Key Ingredients

Here is how their system works, broken down with simple analogies:

1. The "Translator" (The Large Language Model)

Imagine you are a non-technical manager who says to your team: "I want to see every time a red truck passes a school bus in the rain."

Old Way: You'd have to hire a programmer to write hundreds of lines of code to define "red," "truck," "school bus," and "rain," and then tell the computer how to check for them.
New Way: You just type that sentence into the system. An AI (a Large Language Model) acts as a translator. It takes your English sentence and instantly turns it into a strict, formal set of instructions.

2. The "Rulebook" (The Domain-Specific Language or DSL)

This is the most important part. The AI doesn't just write random code; it writes instructions using a special, simplified "Rulebook" called a DSL.

The Analogy: Think of the DSL as a Lego set with only the specific pieces you need. You can't build a random house; you can only build specific, pre-approved structures.
Why this matters: Because the AI is forced to use this "Lego set," it can't make mistakes or write messy code. It ensures every instruction is:
- Safe: It won't accidentally crash the car's computer.
- Fast: It's optimized to run on the car's small, weak processor.
- Checkable: You can easily read the "Lego structure" to see if it makes sense.

3. The "Bouncer" (The Trigger Framework)

Once the AI translates your request into the DSL "Lego structure," it gets sent to the car's computer. This acts like a bouncer at a club.

The bouncer stands at the door (the data stream).
He has your specific rule: "Only let in red trucks near school buses in the rain."
As the car drives, the bouncer checks every second of data. If the condition isn't met, he ignores it. If the condition is met, he opens the door and saves that specific clip.

Why is this better than the old way?

The paper tested this against two other methods:

Writing raw code directly: This is flexible but messy. If you ask the AI to write raw code, it might write 10 different versions of the same rule, some of which are slow or buggy. It's like asking a chef to cook a meal without a recipe; sometimes it's great, sometimes it's a disaster.
Using a Vision-Language Model (VLM): This is like asking a human to watch the video and decide what to save. It's very smart but incredibly slow and expensive to run on a car.

The Results:

Consistency: The DSL method was like a factory assembly line. Every time you asked for "red trucks," the system built the exact same, perfect filter.
Speed: Because the rules were simplified (like a pre-made Lego set), the car could check them much faster than the other methods.
Accuracy: It found the right data just as well as the complex methods, but without the headache.

The Bottom Line

This paper introduces a system where you can tell a car, "Save me the funny moments," or "Save me the scary moments," in plain English. The system then automatically builds a super-fast, reliable filter to catch exactly those moments, saving money on storage and making it easier for engineers to build better self-driving cars.

It turns data collection from a "record everything and hope for the best" approach into a "ask for what you need and get exactly that" approach.

1. Problem Statement

Data-driven systems, particularly in autonomous driving and robotics, rely heavily on high-quality, task-relevant data. However, current data collection pipelines suffer from three critical inefficiencies:

Indiscriminate Logging: Continuous logging of all multimodal sensor streams (cameras, LiDAR, telemetry) creates massive storage costs and captures vast amounts of irrelevant data.
Expert Dependency: Existing selective capture methods rely on hand-written, low-level code. This requires deep expertise in signal interfaces and system architecture, making trigger development a bottleneck that is difficult to audit, reuse, or scale.
Lack of Modularity and Intent Alignment: Current triggers are often monolithic, inefficiently composed, and equate "relevance" with generic anomaly signals rather than specific user intents (e.g., "collect data when a pedestrian is in the rain"). This leads to missed events, duplicated captures, and a disconnect between stakeholder requirements and deployable logic.

The core challenge is bridging the gap between high-level natural language requirements (e.g., "collect rare pedestrian interactions in urban rain") and efficient, verifiable, on-device trigger execution across heterogeneous sensors.

2. Methodology

The authors propose a declarative framework that combines Large Language Models (LLMs) with a formally specified Domain-Specific Language (DSL) to automate trigger generation.

A. The Domain-Specific Language (DSL)

The framework introduces a DSL defined via Extended Backus-Naur Form (EBNF) to formalize trigger logic.

Formal Definitions: The system defines a Scene ( $S$ ), Vehicle State ( $V$ ), Situation ( $S \cup V$ ), Predicate ( $P$ ), and Trigger ( $T$ ).
Structure: A trigger is a Boolean function combining atomic predicates (e.g., object detection, distance checks, attribute filtering) using logical operators (AND, NOT).
Syntax: The DSL uses keywords like TRIGGER, PREDICATE, and AS to define conditions. For example, a predicate might check if a detected object is a "person" within a specific distance.
Output: The DSL compiles into a structured JSON representation that serves as input for the execution engine, ensuring syntactic correctness and modularity.

B. LLM Integration (RAG)

To automate the creation of these DSL programs:

Natural Language to DSL: An LLM translates user requests (natural language) into valid DSL constructs.
Retrieval-Augmented Generation (RAG): To ensure accuracy, the LLM is augmented with system-specific knowledge (e.g., signal descriptions, sensor mappings, codebases) retrieved from unstructured data sources.
Process: The LLM generates a JSON configuration adhering strictly to the DSL grammar, which is then deployed to the vehicle.

C. In-Vehicle Trigger Architecture

The framework operates on a resource-constrained edge platform (ROS environment) with three main components:

Data Manager: Acts as the data backbone. It filters, preprocesses, and caches relevant vehicle signals (creating a "current situation" snapshot) to feed the trigger engine.
Trigger Engine: A dedicated instance for each trigger. It evaluates the logical condition ( $T$ ) against the cached data in real-time. It only executes when new data arrives and frequency criteria are met.
Trigger Manager: Orchestrates multiple trigger engines, managing evaluation schedules, resource constraints, and aggregating results (via an OR-gate logic) to initiate data logging.

3. Key Contributions

A Formal DSL for Multimodal Triggers: A language that supports conditional logic and composition, enabling formal verification and efficient execution on edge devices.
LLM-Driven Automation: A framework that leverages LLMs to translate high-level user intents into auditable, reusable capture programs, reducing the need for specialized coding expertise.
Empirical Validation: A comprehensive evaluation on vehicular and robotic platforms demonstrating that the DSL approach outperforms unconstrained code generation in consistency and latency while maintaining detection performance.

4. Experimental Results

The authors evaluated the framework against two baselines: Unconstrained Code Generation (LLM writing raw Python code) and a Vision-Language Model (VLM) approach (direct inference without triggers).

Detection Performance:
- The DSL approach achieved comparable detection performance to unconstrained code generation (F1 scores around 0.66–0.76).
- The VLM approach showed the highest F1 score (0.67) but lacked scalability.
- Note: Detection accuracy is limited by the underlying perception models (YOLO, CLIP), not the trigger framework itself.
Generation Consistency:
- DSL: Demonstrated significantly higher consistency across multiple generation runs (Sequence similarity: 0.966 vs. 0.392 for code). The structured grammar prevents the syntactic variations common in raw code generation.
- Code: Showed high variability in character-level structure despite similar semantic intent.
Runtime Performance:
- DSL: Achieved the lowest execution latency. The structured, optimized logic allows for efficient evaluation on embedded hardware.
- Code: Often introduced redundant checks, reducing efficiency.
- VLM: Had the highest latency due to the computational cost of running inference for every query, making it unsuitable for high-frequency real-time triggering.

5. Significance and Conclusion

This work represents a paradigm shift from passive logging to intent-driven, active data collection.

Scalability & Modularity: By abstracting trigger logic into a DSL, the system allows for modular composition. New triggers can be added or modified without redesigning the entire pipeline, addressing the "bottleneck" of manual trigger development.
Edge Efficiency: The framework is specifically designed for resource-constrained edge platforms, offering lower latency and predictable execution compared to heavy VLM inference or unoptimized code.
Verifiability: The formal grammar ensures that generated triggers are syntactically correct and auditable, a crucial requirement for safety-critical systems like autonomous vehicles.

Limitations & Future Work:
The system's detection capability is still bound by the performance of the underlying perception models (e.g., YOLO/CLIP). Future work aims to combine the structural guarantees and efficiency of the DSL with the creative flexibility of unconstrained code generation, potentially using adaptive reasoning mechanisms to handle complex, non-standard scenarios.