💬 NLP

FPMoE: A Sparse Mixture-of-Experts Approach to Functional Code Generation

FPMoE is a lightweight, open-source code generation model that leverages a sparse Mixture-of-Experts architecture with dedicated and shared experts to overcome the limitations of existing models in functional programming languages, achieving superior performance on Haskell, OCaml, and Scala while matching much larger models with only 3B active parameters.

Original authors: Loc Pham, Lang Hong Nguyet Anh, Thanh Le-Cong

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Loc Pham, Lang Hong Nguyet Anh, Thanh Le-Cong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a team of AI assistants how to write code in Functional Programming languages (like Haskell, OCaml, and Scala). These languages are like a specific dialect of math: they are very logical, strict, and different from the "imperative" languages (like Python or Java) that most computers and AI are used to.

The paper, titled FPMoE, argues that current AI models are terrible at this specific dialect. Here is the simple breakdown of the problem, the solution, and why it works, using everyday analogies.

The Problem: The "One-Size-Fits-All" vs. "Specialist" Dilemma

The researchers tried two standard ways to fix the AI, and both failed:

The "Specialist" Approach (Per-Language Fine-Tuning):
- The Idea: Train one AI just for Haskell, another just for OCaml, and a third just for Scala.
- The Failure: It's like hiring three separate chefs who only know how to cook one specific dish. They become great at that one dish, but they forget the universal rules of cooking (like how heat affects ingredients) that apply to all dishes. They miss the big picture.
The "Generalist" Approach (Multi-Language Fine-Tuning):
- The Idea: Train one single AI on all three languages mixed together.
- The Failure: It's like putting three different dialects of English into one person's brain at the same time. The person gets confused, mixing up the rules. They end up writing code that looks like a messy hybrid, failing to sound natural in any of the three languages. This is called "cross-language interference."

The Solution: The "FPMoE" Team

The authors created a new model called FPMoE (Functional Programming Mixture-of-Experts). Think of this not as one giant brain, but as a specialized team of experts working together in a single office.

The team consists of four members:

Three Language Specialists (The Routed Experts):
- There is one expert dedicated only to Haskell, one only to OCaml, and one only to Scala.
- How they work: When the AI needs to write Haskell code, a "manager" (called a router) sends the task only to the Haskell expert. This ensures the AI doesn't accidentally mix in OCaml rules. This solves the "confusion" problem.
One Universal Mentor (The Shared Expert):
- This is a fourth expert who is always active, no matter which language is being used.
- What they do: This expert knows the deep, shared logic of functional programming (like "monadic reasoning," which is a fancy way of saying "how to handle complex steps in a logical chain").
- Why it matters: Even though the specialists know their specific language, they might forget the universal rules of functional logic. The Universal Mentor is always there to whisper, "Remember, in functional programming, we don't change things; we transform them." This ensures the code follows the correct style, not just the correct syntax.

Why It's a Big Deal

The paper claims this "Team Approach" is a magic trick for efficiency:

Small but Mighty: The FPMoE model only "wakes up" about 3 billion parameters (brain cells) at a time to do a task.
Beating Giants: Despite being small, it performs just as well as massive models that have 14 billion or even 30 billion parameters.
The Analogy: Imagine a small, highly organized team of three specialists and one mentor can solve a puzzle just as fast as a giant, chaotic crowd of 30 people.

The Results

When tested on a benchmark called FPEval (a test of how well AI can write functional code):

Better Accuracy: FPMoE wrote code that actually worked much more often than the previous methods.
Better Style: It didn't just write code that passed the test; it wrote code that looked like it was written by a human functional programmer (avoiding "imperative" habits).
Efficiency: It achieved these results using a fraction of the computing power required by larger models.

The Catch (Limitations)

The paper is honest about what this model can't do yet:

Fixed Team: The team is hardcoded for exactly three languages (Haskell, OCaml, Scala). If you want to add a fourth language (like Clojure), you can't just "hire" a new expert; you have to rebuild the whole team from scratch.
Memory: Even though the model only uses a small amount of brain power at any moment, the entire team's knowledge must be loaded into the computer's memory at once, which can be heavy for some computers.

Summary

FPMoE solves the problem of AI struggling with functional programming by stopping the AI from trying to be one confused generalist. Instead, it gives the AI a specialized team: three experts who know their specific languages perfectly, and one mentor who ensures they all stick to the core rules of functional logic. This allows a small model to punch way above its weight class.

Technical Summary: FPMoE – A Sparse Mixture-of-Experts Approach to Functional Code Generation

1. Problem Statement

Despite rapid advancements in Large Language Models (LLMs) for code generation, functional programming languages (FPLs) such as Haskell, OCaml, and Scala remain chronically underexplored. Existing frontier models perform significantly worse on FPLs compared to imperative languages like Java, often generating code that is functionally correct but violates idiomatic functional style (e.g., failing to use monads or referential transparency).

The paper identifies a structural data imbalance: the FPL developer community is small, producing orders of magnitude less open-source code than mainstream languages. Consequently, scaling pre-training data alone is insufficient to close the performance gap.

The authors investigate fine-tuning as a remedy but find two distinct failure modes in standard approaches:

Per-language fine-tuning: Training separate models for each language fails to capture shared functional abstractions (e.g., monadic reasoning) that generalize across languages.
Multi-language fine-tuning: Training a single dense model on a mixture of languages introduces cross-language interference, degrading performance on individual languages (particularly OCaml).

The core challenge is to simultaneously preserve language-specific specialization and shared cross-language generalization.

2. Methodology

The authors propose FPMoE, a lightweight, open-source code generation model built on a sparse Mixture-of-Experts (MoE) architecture. The training pipeline consists of three stages:

2.1 Data Collection and Pre-processing

Corpora: Data is sourced from the CodeParrot GitHub crawl and the OCaml Corpus, filtered for Haskell, Scala, and OCaml repositories.
Filtering: Files with excessive line lengths (auto-generated/minified code) are discarded.
Deduplication: Performed at the repository level to maintain structural integrity.
Decontamination: N-gram filtering excludes training data containing substrings found in the FPEval test set.
Sequence Construction: Training sequences are constructed at the repository level (concatenating all source files in a repo) rather than the file level to preserve intra-repository dependencies and long-range context.

2.2 Expert Pre-training

Two parallel supervised fine-tuning (SFT) runs are performed on a base model:

Language-specific SFT: The base model is fine-tuned independently on each of the three language corpora, creating three specialized checkpoints.
Mixed-language SFT: A separate instance is fine-tuned on the union of all three corpora. This serves as both a dense baseline and the initialization for the shared expert.

2.3 Joint MoE Fine-tuning

The final model is assembled by replacing the Feed-Forward Network (FFN) sublayers of the transformer backbone with a specialized MoE-FFN block:

Routed Experts: Three experts ( $E_1, E_2, E_3$ ) are initialized from the language-specific checkpoints. A lightweight router uses Top-K selection (K=2) to dispatch tokens to the appropriate language expert.
Shared Expert: A fourth expert ( $E_{shared}$ ) is always active, regardless of the routing decision. It is initialized from the mixed-language checkpoint to encode cross-language functional patterns (e.g., type-directed programming).
Training Objective: The assembled model is trained jointly on the combined corpus using a causal language modeling objective augmented with an auxiliary load-balancing loss to prevent routing collapse.

3. Key Contributions

Problem Analysis: An empirical demonstration that standard fine-tuning strategies (per-language vs. multi-language) exhibit complementary failure modes that cannot be resolved simultaneously by dense models.
Model Architecture (FPMoE): The introduction of a sparse MoE architecture that explicitly incorporates language identity as an inductive bias for routing. Unlike prior data-driven MoE approaches (e.g., Mixtral, DeepSeekMoE), FPMoE initializes routed experts with language-specific knowledge and employs a dedicated, always-active shared expert to capture cross-language abstractions.
Evaluation and Analysis: Beyond functional correctness, the authors evaluate FPViolation (adherence to idiomatic style). They provide a controlled ablation study decomposing the gains of the architecture into contributions from language-aligned routing and the shared expert.

4. Experimental Results

Experiments were conducted on the FPEval benchmark (721 tasks across Haskell, OCaml, and Scala).

Performance vs. Dense Baselines: FPMoE substantially outperforms dense fine-tuned baselines.
- The FPMoE-Qwen2.5-4x1.5B-A3B (3B active parameters) achieves Pass@1 scores of 7.37% (Haskell), 6.8% (OCaml), and 13.31% (Scala). This represents a 5–11× improvement over its dense 1.5B counterpart.
- The FPMoE-Qwen2.5-4x3B-A6B (6B active parameters) achieves 11.00% (Haskell), 12.34% (OCaml), and 21.08% (Scala).
Parameter Efficiency: The 6B-active-parameter FPMoE model matches or surpasses much larger dense models, including DeepSeek-Coder-6.7B and Qwen2.5-Coder-14B, while activating less than half their parameters. It also surpasses GPT-3.5 on OCaml and Scala.
Idiomaticity: FPMoE demonstrates a better balance between correctness and adherence to functional style (lower FPViolation rates) compared to larger frontier models, which often generate imperative-style code that passes tests but violates functional principles.

Ablation Study Findings

Shared Expert Initialization: Initializing the shared expert from a mixed-language corpus (Config A) is critical. Initializing from a general base model (Config B) or freezing it (Config C) results in performance drops, sometimes below dense baselines.
Necessity of Shared Expert: Removing the shared expert (Config D) reduces performance, confirming its role in capturing cross-language abstractions.
Language-Aligned Routing: Removing language-specific warm-up (Config E) causes a significant performance drop, particularly on OCaml, proving that the router cannot discover specialization purely from data without the inductive bias.

5. Significance and Claims

The paper claims that targeted architectural specialization offers a more principled path to functional code generation than brute-force data scaling. The authors argue that:

The FPL data ecosystem is structurally small and unlikely to grow fast enough to support massive pre-training alone.
Frontier models trained on larger corpora still exhibit high rates of idiomatic violations, suggesting that data scaling alone does not solve the problem.
FPMoE's design effectively utilizes available FPL tokens by separating language-specific syntax (via routed experts) from shared functional logic (via the shared expert).

The authors position FPMoE as a step toward lowering the barrier to entry for functional programming, though they acknowledge limitations such as the fixed set of supported languages, potential memory footprint issues during inference, and the recency of the training data.