Constructing Synthetic Instruction Datasets for Improving Reasoning in Domain-Specific LLMs: A Case Study in the Japanese Financial Domain

This study presents a general method for constructing high-quality synthetic instruction datasets with Chain-of-Thought reasoning traces starting from domain-specific vocabulary, demonstrating its effectiveness in enhancing reasoning capabilities for Japanese financial domain LLMs through a 9.5 billion-token dataset.

The Gist

Imagine you want to teach a brilliant but generalist student (an AI) how to become a top-tier financial advisor in Japan. You can't just hand them a dictionary of finance terms; they need to learn how to think like an expert, not just what to say.

This paper describes a clever recipe for creating a massive, custom-made "training manual" for AI, specifically designed to teach it how to reason through complex Japanese financial problems.

Here is the breakdown of their method, using some everyday analogies:

1. The Problem: The "Smart but Clueless" Student

Current AI models are like students who have read every book in the library but have never taken a math test or solved a real-world case. They know facts, but when you ask them, "How do I predict if this company will go bankrupt?" they often guess or give a shallow answer. They lack the Chain of Thought (CoT)—the internal monologue where an expert breaks a problem down step-by-step before answering.

2. The Solution: Building a "Reasoning Gym"

The authors didn't just feed the AI more books. They built a synthetic training gym (a dataset) where the AI practices thinking before speaking.

Step A: Picking the Seeds (Topic Selection)

Instead of guessing what to teach, they started with a list of "seed words" like Insurance, Securities, and Financial Planning.

• Analogy: Imagine planting a garden. You don't just throw seeds in the dirt; you carefully select specific seeds (topics) that represent the whole garden. They also mixed in some "wildflowers" (general topics) so the AI didn't forget how to talk about normal things while learning finance.

Step B: Growing the Questions (Instruction Generation)

They used a super-smart AI to generate thousands of questions based on those seeds.

• Analogy: It's like a master chef asking a sous-chef to invent 10,000 new recipes based on "tomatoes." Some recipes are simple (open-ended questions), some are math-heavy (calculations), and some are creative (writing reports).

Step C: The "Remix" Station (Expansion & Modification)

They took those questions and tweaked them. They changed the format, added context, or turned a simple question into a complex scenario.

• Analogy: If the original question was "What is an interest rate?", the remix station turns it into, "If I have $10,000 and the rate changes due to a new law, how does my savings grow?" This forces the AI to handle different angles of the same topic.

Step D: The Quality Control (The "Judge")

Not all generated questions are good. Some might be nonsense or too easy. They used an AI "Judge" to grade the data.

• Analogy: Imagine a strict editor reviewing a stack of student essays. If an essay doesn't show the work (the reasoning steps) or has a wrong answer, the editor throws it in the trash. Only the essays that show clear, logical thinking make the cut.

3. The Result: A Massive Library of Thought

They ended up with a dataset containing 9.5 billion tokens (a massive amount of text) where every single example includes the "thinking process" (the reasoning trace) before the final answer.

They trained their AI models on this data and found:

• Better Scores: The AI got significantly better at financial exams and tasks compared to models that just memorized facts.
• The "Thinking" Matters: When the AI was forced to "think out loud" (generate reasoning traces) before answering, it got the right answers much more often.

4. The Twist: The "Goldilocks" Zone of Thinking Length

One of the most interesting findings was about how long the AI should think.

• Too Short: The AI rushes to the answer and makes mistakes.
• Just Right (1,024 tokens): The AI has enough time to break down the problem, check its work, and find the right answer. Performance peaks here.
• Too Long (4,096+ tokens): The AI starts to get bored or confused. It starts repeating itself or looping in circles, saying "Wait, let me think..." over and over without actually solving the problem.
• Analogy: It's like a student taking a test. If they have 5 minutes, they panic. If they have 30 minutes, they do great. If you give them 5 hours, they might start staring at the ceiling, doodling, or rewriting the same sentence 50 times, and their score actually drops.

5. Why This Matters

This paper proves that you don't need a human to write millions of examples to train a specialist AI. You can use AI to generate its own high-quality "thinking practice" data.

• The Takeaway: To make AI smart in a specific field (like finance, law, or medicine), you don't just need more data; you need better thinking patterns. By teaching the AI how to reason step-by-step, you turn a generic smart assistant into a true domain expert.

The authors have even shared their "gym" (the dataset and models) for others to use, hoping to help build smarter AI for all kinds of specialized jobs.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) are increasingly deployed in society, adapting them to high-expertise domains like finance and law remains challenging. Two critical gaps exist:

• Knowledge vs. Reasoning: Traditional adaptation methods (Continued Pre-Training) effectively acquire domain knowledge but often fail to develop the advanced logical reasoning required for complex tasks (e.g., fraud detection, earnings forecasting).
• Data Scarcity: While general-purpose reasoning datasets exist, there is a lack of large-scale, domain-specific instruction datasets (particularly in Japanese) that include Chain-of-Thought (CoT) reasoning traces. Existing Japanese financial datasets often lack reasoning traces or are limited in scale.

2. Methodology

The authors propose a general, domain-agnostic pipeline for constructing high-quality synthetic instruction data starting from domain-specific vocabulary. The pipeline consists of seven stages:

1. Domain Topic Word Selection:

   • Seed words (industries, products, technologies) are selected for the target domain (Finance).
   • To prevent "Catastrophic Forgetting" of general capabilities, seed words from non-target domains are mixed in.
   • Sub-topic words are generated for each seed.

2. Instruction Generation:

   • An LLM generates user prompts (instructions) based on sub-topics.
   • Four task types are defined to ensure diversity: Open-ended questions, Mathematical reasoning, Creative writing, and Multiple-choice questions.

3. Instruction Expansion and Modification:

   • Instructions are expanded by adding context, changing formats/styles, elaborating on specific cases, and rewriting for related topics.
   • Variants are generated proportionally to task types.

4. Filtering and Deduplication:

   • N-gram and Word Count Filtering: Removes extremely short or low-quality instructions (using MeCab for Japanese tokenization).
   • Deduplication: Uses MinHash and LSH (Locality Sensitive Hashing) for fuzzy deduplication, following the Swallow Corpus v2 methodology.

5. Multi-turn Conversation Generation:

   • Single-turn instructions are converted into multi-turn dialogues to enhance interactive reasoning.
   • The first turn uses the original instruction; subsequent turns alternate between user prompts and LLM responses containing reasoning traces.

6. Quality Filtering (LLM-as-a-Judge):

   • A high-capability LLM (gpt-oss-120b) acts as a judge to evaluate whether data accurately reflects domain knowledge and contains logical reasoning.
   • Low-quality data is excluded based on average scores across multiple criteria.

7. Integration with Public Datasets:

   • Public datasets (NuminaMath-CoT, Nemotron-Post-Training-Dataset-v1, smol-constraints) are integrated to reinforce fundamental math and instruction-following abilities. Only question portions are extracted and processed.

Dataset Scale: The final dataset contains approximately 1.44 million samples totaling 9.5 billion tokens.

3. Experimental Setup

• Base Models: Qwen3-14B-Base and gpt-oss-20b.
• Training Strategy:
  1. Continued Pre-Training (CPT): Models were pre-trained on a filtered Japanese financial corpus (derived from Common Crawl).
  2. Supervised Fine-Tuning (SFT): Models were fine-tuned on the constructed synthetic dataset (2 epochs, 8x NVIDIA H200 GPUs).
• Evaluation Benchmarks:
  • japanese-lm-fin-harness: Tests financial knowledge (sentiment analysis, certification exam questions).
  • pfmt-bench-fin-ja: Tests financial dialogue capabilities (multi-turn composition, information extraction).

4. Key Results

• Performance Improvement:
  • Models trained with the proposed pipeline (CPT + SFT) surpassed official instruction-tuned models on financial benchmarks.
  • gpt-oss-20b + CPT + SFT achieved the highest scores, outperforming the official baseline on all subtasks.
  • Qwen3-14B + CPT + SFT showed similar trends, significantly improving over the base model.
• Impact of Reasoning Traces (CoT):
  • Enabling CoT inference improved performance by 4.5–5.7 points on japanese-lm-fin-harness and ~0.4 points on pfmt-bench-fin-ja across all models.
• Reasoning Trace Length Analysis:
  • Optimal Length: Performance improved as reasoning trace length increased up to 1,024 tokens.
  • Diminishing Returns: Extending traces beyond 2,048 tokens yielded no significant improvement.
  • Failure Modes:
    • Early Termination: Models often inserted "Wait," but immediately concluded without deep reasoning.
    • Output Loops: At lengths >4,096 tokens, models entered loops repeating conclusions to fill space without adding value.
  • Forced Termination Penalty: Artificially truncating reasoning traces (to enforce specific lengths) caused a performance drop of several points compared to natural termination, suggesting the method of termination matters.

5. Key Contributions

1. Generalizable Framework: Established a scalable pipeline for synthesizing domain-specific instruction data with CoT traces, applicable to any high-expertise domain (demonstrated here on Japanese Finance).
2. Large-Scale Japanese Financial Dataset: Created a 9.5B token dataset with reasoning traces, filling a critical gap in Japanese domain-specific resources.
3. Empirical Validation of Reasoning: Demonstrated that combining domain knowledge injection (CPT) with reasoning-focused SFT maximizes task performance.
4. Trace Length Insights: Provided empirical evidence that reasoning traces have an optimal length (approx. 1k tokens) in domain-specific tasks, beyond which performance plateaus or degrades due to model behaviors like looping.

6. Significance

This study bridges the gap between domain knowledge acquisition and logical reasoning capability in LLMs. It proves that synthetic data generation, starting from topic words and filtered via LLM-as-a-Judge, is a highly effective strategy for building specialized models. The findings on reasoning trace length offer crucial guidance for inference optimization in financial and other regulated domains, suggesting that "more reasoning" is not always better and that natural termination is superior to forced truncation. The authors have open-sourced their models and datasets on HuggingFace to facilitate further research.