Leveraging LLMs for Multi-File DSL Code Generation: An Industrial Case Study
This industrial case study at BMW demonstrates that fine-tuning code-oriented LLMs with a structured, path-preserving JSON representation of repository hierarchies significantly outperforms baseline prompting and one-shot learning in generating accurate, multi-file DSL artifacts that drive downstream code generation.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a master architect working for a massive construction company (BMW). Your job is to design new buildings (market configurations) for different cities.
In the old way of doing things, you had to manually write out every single blueprint, electrical plan, and plumbing diagram on separate pieces of paper. If you wanted to add a new window to the kitchen, you had to remember to update the electrical plan, the plumbing plan, and the structural plan, all in perfect sync. If you missed one tiny detail, the whole building couldn't be built, and the construction crew (the code generator) would get stuck.
This paper is about teaching a super-smart robot assistant (a Large Language Model, or LLM) to do this manual paperwork for you. But there's a catch: the robot has never seen your specific blueprints before, and it has to update multiple interconnected papers at once based on just one sentence of instructions from you.
Here is how the researchers taught the robot to do it:
1. The Problem: The "One Instruction, Many Files" Puzzle
Usually, AI is good at writing a single story or a single piece of code. But in this company, a single request like "Add a new loan type for the German market" requires changing files in five different folders. If the robot changes the main file but forgets to update the file that lists all the loan types, the whole system breaks.
2. The Solution: Flattening the Library
To help the robot understand the whole project at once, the researchers didn't feed it file by file. Instead, they took the entire "library" of blueprints (the folder structure) and flattened it into one giant, organized book (a single JSON document).
- The Analogy: Imagine taking a messy desk with 20 different folders and laying every single page out on one giant table, but keeping them in neat rows so you know which page belongs to which folder. Now, the robot can look at the whole table at once and say, "Okay, I see where the changes need to go."
3. The Training: Three Ways to Teach the Robot
The researchers tried three different ways to teach the robot this specific job:
- The "Cold Start" (Zero-Shot): They just gave the robot the instruction and the flattened book and said, "Fix it."
- Result: The robot was okay at keeping the pages in the right order, but it often made up wrong details or missed tiny connections. It was like a student who knows the alphabet but hasn't learned the grammar yet.
- The "Example" (One-Shot): They gave the robot the instruction, the book, plus one example of a previous job done correctly. "Here is how we added a window last time; do it like this."
- Result: This helped the robot get the structure right (the pages stayed in the right folders), but it still struggled with the specific, tricky details of the new request.
- The "Intensive Course" (Fine-Tuning): They took the robot and showed it hundreds of examples of "Before" and "After" blueprints. They didn't just show it; they adjusted the robot's internal brain (using a technique called QLoRA) so it learned the specific rules of this company's blueprints.
- Result: This was the winner. The robot became an expert. It could look at a request and produce a perfectly synchronized set of changes across all files.
4. The Results: Did It Work?
The researchers tested the robot with real tasks and had human experts check the work.
- Structure: The fine-tuned robot was almost perfect at keeping the files in the right folders and not inventing fake folders. It got a perfect score (1.0) on keeping the "shape" of the project correct.
- Details: It was very good at the actual content, getting about 66% of the changes exactly right and 84% of the changes "close enough" to be useful.
- The "Near-Miss" Reality: The robot was so good that when it failed, it was usually just a tiny typo or a wrong number, not a missing file. It was like a chef who cooks a perfect meal but accidentally uses salt instead of sugar. The structure was right; the flavor was just slightly off.
5. What the Human Experts Said
The researchers asked four senior architects (developers) to review the robot's work.
- Verdict: They loved it. They gave it high marks for readability and structure.
- Time Savings: They estimated that for small, simple projects, the robot could save them 40% to 80% of their time. However, for massive, complex projects, the robot was still a "helper" that needed a human to double-check the final details.
- The Catch: The robot sometimes got confused about which specific type of part to use in a complex scenario, requiring a human to make a quick fix.
The Bottom Line
This paper proves that you can teach an AI to manage complex, multi-file projects if you:
- Organize the files into a single, easy-to-read format.
- Give it a specific "intensive course" (fine-tuning) on your specific rules.
It won't replace the human architects yet, but it can do the heavy lifting of drafting the blueprints, saving them hours of tedious work and letting them focus on the creative parts.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.