💬 NLP

LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance

This paper employs perturbation-based attribution analysis to reveal that full fine-tuning yields more focused interpretive behaviors than parameter-efficient methods, while increasing model scale enhances attention to specific rule constraints despite performance gains plateauing beyond 7B parameters, thereby advancing the explainability of LLMs for automated code compliance in the AEC industry.

Original authors: Jack Wei Lun Shi, Minghao Dang, Wawan Solihin, Justin K. W. Yeoh

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jack Wei Lun Shi, Minghao Dang, Wawan Solihin, Justin K. W. Yeoh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but somewhat mysterious, robot how to read a building code book and turn those rules into a computer program. This is what the paper is about: teaching robots to check if buildings follow the rules (like "fire escapes must be 2 meters wide") and turning those text rules into code that a computer can understand.

The researchers wanted to know: How does the robot actually "think" when it does this? Does it read the whole sentence carefully, or does it just guess based on a few keywords? And does it matter how we teach the robot (the training method) or how big the robot's brain is (the model size)?

Here is the breakdown of their findings using some everyday analogies:

1. The Three Ways to Train the Robot (Fine-Tuning Strategies)

Imagine you have a brilliant student (the AI model) who knows a lot about everything. You want to teach them specifically about building codes. You have three ways to do this:

Full Fine-Tuning (FFT): This is like hiring a personal tutor who rewrites the student's entire textbook from scratch, changing every single note and highlight to fit the new subject. It's expensive and takes a lot of time, but the student learns the material perfectly.
- The Result: The robot became very focused. It knew exactly which words in the rule were the "keys" to the solution. It didn't waste time guessing; it zeroed in on the critical numbers and specific terms.
LoRA (Low-Rank Adaptation): This is like giving the student a set of sticky notes and a highlighter. You don't rewrite the whole book; you just add notes to the margins. It's cheaper and faster.
- The Result: The robot did a good job, but its "thinking" was a bit more scattered. It looked at more words, unsure of which ones were truly the most important.
QLoRA (Quantized LoRA): This is like giving the student the sticky notes, but the ink is very faint (compressed). It's even cheaper, but some details get blurry.
- The Result: The robot struggled the most. It was the least confident about which words mattered, often spreading its attention too thin.

The Takeaway: If you want the robot to be a precise surgeon, you need the "Full Fine-Tuning" method. If you just want a quick check-up, the "sticky note" methods work, but the robot might be a bit fuzzy on the details.

2. The Size of the Brain (Model Scale)

The researchers also tested robots with different brain sizes: small (3 billion parameters), medium (7 billion), and huge (22 billion).

The Small Brain (3B): Imagine a junior intern. They read the rule and highlight almost everything. They are unsure of what's important, so they highlight "the," "and," "must," and "height" all with equal intensity. They are trying to be safe by not missing anything.
The Medium Brain (7B): This is a senior engineer. They start to ignore the fluff. They focus on the numbers and the specific rule names.
The Giant Brain (22B): This is a master architect. They don't just read the words; they understand the structure of the rule.
- The Magic Trick: The giant brain realized that the Rule ID number (like "Section 4.3.11") is actually a super-powerful clue. It also learned that the exact number (like "2.2 meters") is the most critical part of the sentence.
- The Catch: Once the brain got bigger than 7 billion, making it even bigger didn't make it better at the average task. It was like buying a Ferrari when a Toyota was already fast enough; the extra speed didn't help you get to the grocery store any faster. However, the giant brain was better at handling the hardest cases.

3. The "Black Box" Problem

Usually, when we use AI, we treat it like a black box: we put a rule in, and a code comes out. We don't know why it made that code.

This paper is like putting a transparent window on that black box. By using a technique called "attribution analysis" (which is like shining a flashlight on the words the robot is looking at), the researchers could see:

Did the robot focus on the right words?
Did it get confused by the wrong words?
Did it learn to trust the rule numbers?

Why Does This Matter?

In the construction industry, if a robot makes a mistake, a building could be unsafe. We can't just trust a robot because it "got the right answer" 90% of the time. We need to know how it got there.

If the robot is focusing on the wrong words (like focusing on the color of the text instead of the numbers), it might pass a dangerous building.
This study shows that to build trustworthy robots for safety-critical jobs, we need to use the right training method (Full Fine-Tuning) and understand that bigger isn't always better—it's about finding the "Goldilocks" size where the robot is smart enough to be precise but not so big that it's wasteful.

In a nutshell: The researchers figured out that to make AI reliable for building safety, we need to train it thoroughly (not just with sticky notes) and teach it to focus on the specific numbers and rule IDs, rather than just guessing based on the general vibe of the sentence.

1. Problem Statement

While Large Language Models (LLMs) show promise in Automated Code Compliance (ACC)—specifically in converting complex building regulations into computer-processable scripts (e.g., Python)—existing research treats these models as "black boxes." Current studies focus almost exclusively on output performance metrics (accuracy, semantic similarity) without investigating how training decisions influence the model's internal interpretive mechanisms.

Key gaps identified include:

Lack of Explainability: It is unclear how different fine-tuning strategies (Full Fine-Tuning vs. Parameter-Efficient Fine-Tuning) affect which parts of a regulation text the model prioritizes.
Model Scale Impact: The influence of model parameter size (e.g., 3B vs. 22B) on the interpretive focus of the model remains unexplored.
Trust Barrier: Without understanding the attribution patterns (what words drive the output), the adoption of LLMs for critical, regulation-bound tasks in the Architecture, Engineering, and Construction (AEC) industry is hindered.

2. Methodology

The study employs a systematic three-stage approach to analyze interpretive behaviors:

A. Experimental Setup

Dataset: A real-world dataset of building regulations paired with corresponding Python script implementations.
Variables:
1. Fine-Tuning Strategies: Full Fine-Tuning (FFT), Low-Rank Adaptation (LoRA), and Quantized LoRA (QLoRA).
2. Model Scales: Ranging from 0.5B to 22B parameters (using families like Qwen, LLaMA, Falcon, Mistral, etc.).
Selection for Attribution Analysis: Due to computational constraints, attribution analysis was performed on a representative subset:
- Strategy Comparison: LLaMA 3B (FFT, LoRA, QLoRA).
- Scale Comparison: LLaMA 3B, Mistral 7B, and Mistral-Small 22B (all QLoRA).

B. Performance Evaluation

Metric: CodeBERTScore (F1 and F3).
Rationale: Unlike lexical matching, CodeBERTScore uses contextual embeddings to measure semantic similarity between generated and reference scripts, correlating better with functional correctness.

C. Perturbation-Based Attribution Analysis

Tool: Captum library (PyTorch).
Technique: Feature Ablation. The method systematically removes (ablates) individual words from the input building rule and measures the change in the log-probability of the target script generation.
Attribution Score ( $\phi$ ): Calculated as the difference in log probability between the original input and the input with the $i$ $i$ -th word removed.
- Positive Score: The word increases confidence in the correct output.
- Negative Score: The word hinders the output.
- Zero: Negligible effect.
Normalization: Visualizations use normalized color scales to compare attribution magnitudes across different models and strategies fairly.

3. Key Contributions

Shift from Performance to Interpretability: Moves beyond "does it work?" to "how does it work?" by analyzing the internal attention mechanisms of LLMs in the AEC domain.
Quantitative Comparison of Training Strategies: Provides statistical evidence on how FFT, LoRA, and QLoRA alter the model's focus on textual features.
Analysis of Scaling Laws in Interpretation: Investigates how increasing model parameters changes the specific textual features (e.g., rule IDs, numerical constraints) that models prioritize.
Methodological Framework: Establishes a perturbation-based attribution workflow specifically tailored for code generation tasks from regulatory text.

4. Key Results

A. Performance vs. Strategy

FFT achieved the highest semantic similarity scores, followed by LoRA, with QLoRA performing the worst.
Quantization Impact: The performance gap between LoRA and QLoRA suggests that 4-bit quantization causes significant fidelity loss, particularly in complex rule interpretation tasks.

B. Performance vs. Model Scale

Scaling Trend: Performance improved as model size increased from 0.5B to ~7B.
Plateau: Beyond 7B parameters, performance gains plateaued. Larger models (12B, 22B) did not show significant median improvements over 7B models when constrained by QLoRA, suggesting diminishing returns or that the task complexity is adequately handled by ~7B models.

C. Attribution Patterns (Fine-Tuning Strategies)

Statistical Difference: FFT produced attribution patterns statistically distinct from PEFT methods (LoRA/QLoRA).
Focus vs. Diffusion:
- FFT: Exhibited concentrated attribution, focusing sharply on critical terms (e.g., "Pumping capacity," "Shadow areas"). This indicates a decisive, confident mapping of features to output.
- LoRA/QLoRA: Showed diffuse attribution, spreading importance across many words. This suggests higher uncertainty or a less refined latent mapping compared to FFT.
Similarity: QLoRA and LoRA patterns were more similar to each other than to FFT (Cosine similarity ~0.45 vs. ~0.29).

D. Attribution Patterns (Model Scale)

3B vs. 7B: Similar attribution patterns (Cosine similarity ~0.35).
22B Distinction: The 22B model showed significantly different patterns compared to smaller models (Cosine similarity ~0.15–0.20).
Evolution of Focus:
- Small Models (3B): Distributed attribution broadly, indicating uncertainty in identifying the single most critical word.
- Large Models (22B): Developed specialized interpretive strategies:
  - Prioritized Rule Identifiers (e.g., "4.3.11d"), recognizing them as predictive features.
  - Prioritized Numerical Constraints (e.g., "2.2 metres") over general concepts.
  - Exhibited sharper, more confident peaks in attribution, isolating core features (e.g., "perimeter") that dictate script logic.

5. Significance and Conclusion

Transparency for Critical Tasks: The study demonstrates that model architecture and training choices fundamentally alter how an LLM "reads" regulations. FFT offers the most decisive interpretive behavior, while PEFT methods may introduce ambiguity.
Scaling Insights: While larger models (22B) do not necessarily generate more accurate code on average (due to the QLoRA plateau), they develop more sophisticated interpretive strategies, such as prioritizing specific numerical values and rule IDs, which are crucial for precise code generation.
Future Direction: The findings advocate for a balance between computational cost and interpretive depth. For critical AEC applications, understanding the attribution pattern is as vital as the output accuracy to ensure regulatory compliance and trust.

In summary, this paper provides a foundational step toward explainable AI in construction, revealing that larger models and full fine-tuning lead to more focused, rule-identifier-aware, and numerically precise interpretive behaviors compared to smaller or quantized models.