Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational Awareness

This paper introduces BD-FDG, a framework that leverages Bloom's Taxonomy and structured knowledge organization to generate a high-quality, cognitively layered dataset for fine-tuning LLMs, successfully adapting the Qwen3-8B model to the complex domain of Space Situational Awareness with significant performance gains while preserving general capabilities.

The Gist

Imagine you have a brilliant, all-knowing librarian named Qwen. This librarian has read almost every book in the world. They can write poetry, solve math puzzles, and explain history better than anyone. However, if you ask them a very specific, high-stakes question about Space Situational Awareness (SSA)—like "How do we track a piece of space debris and calculate a safe avoidance maneuver for a satellite?"—they might stumble. They know the words, but they don't know the rules of the game or the step-by-step engineering logic required to keep satellites safe.

This paper is about teaching this brilliant librarian how to become a Space Safety Expert without turning them into a robot that only knows space and forgets everything else.

Here is the story of how they did it, using simple analogies:

1. The Problem: The "Generalist" vs. The "Specialist"

Think of the original AI (the Generalist) as a master chef who can cook a perfect steak, bake a cake, and make a salad. But if you ask them to perform open-heart surgery, they might try to use a knife they know well, but they lack the specific medical training, the sterile protocols, and the understanding of the human body's complex systems.

In the space world, "cooking" is general knowledge. "Surgery" is Space Situational Awareness (SSA). The AI needs to learn the specific "medical procedures" of space: tracking objects, predicting crashes, and making split-second decisions based on strict engineering rules.

The problem was that the data used to train these AIs was like a random pile of recipe books. It had facts, but it didn't teach the logic of how to think through a crisis.

2. The Solution: The "BD-FDG" Framework

The authors created a new training system called BD-FDG. Think of this as a customized, high-tech training academy for the AI. Instead of just feeding it random facts, they built a curriculum based on three clever ideas:

A. The "Mission Map" (Structured Knowledge)

Imagine trying to learn how to fly a plane by reading random paragraphs about clouds. It's chaotic. Instead, the authors built a Mission Map.

• They organized all the space knowledge like a family tree.
• At the top, you have the big goal (e.g., "Track a satellite").
• Branching down, you have the steps: "Detect," "Predict Path," "Plan Maneuver."
• This ensures the AI doesn't just memorize facts; it learns the story of a mission from start to finish.

B. The "Bloom's Ladder" (Cognitive Layering)

This is the most creative part. They used a famous educational theory called Bloom's Taxonomy, which is like a video game difficulty ladder.

• Level 1 (Remember): "What is a satellite?" (Easy)
• Level 2 (Understand): "Why does it orbit?" (Medium)
• Level 3 (Apply): "Calculate its speed." (Hard)
• Level 4 (Analyze): "Why did this sensor fail?" (Very Hard)
• Level 5 (Evaluate): "Which of these three solutions is safest?" (Expert)
• Level 6 (Create): "Design a new tracking system." (Master)

Most AI training stops at Level 2 or 3. This framework forces the AI to climb all the way to Level 6. They generated 230,000 practice questions that get progressively harder, forcing the AI to learn how to think, not just recall.

C. The "Strict Inspector" (Quality Control)

In the real world, a mistake in space can cost millions of dollars. So, they didn't just let the AI write answers and hope for the best. They built an Automated Inspector.

• This inspector checks every answer against a strict Engineering Rulebook.
• Did the AI use the right technical terms? Yes/No.
• Is the logic sound? Yes/No.
• Is the answer safe? Yes/No.
• If the answer is "fluffy" or wrong, the inspector throws it in the trash. Only the "gold standard" answers make it into the training book.

3. The Result: The "Space-Ready" AI

After this intense training, they tested the new AI (called SSA-LLM-8B) against the old one.

• The Test: They gave them 1,600 difficult space questions.
• The Outcome: The new AI didn't just guess; it dominated.
  • It got 82% of the "Arena Battles" (head-to-head comparisons) against the old AI.
  • Its ability to answer correctly jumped by 144% to 176%.
• The Best Part: It didn't forget how to be a generalist. It could still do math and write code almost as well as before. It became a Specialist who is still a Generalist.

The Big Takeaway

This paper proves that you don't need a supercomputer the size of a city to make an AI smart at a specific job. You just need better training data.

By organizing knowledge like a Mission Map, teaching the AI to climb a Difficulty Ladder, and having a Strict Inspector grade every answer, you can turn a general-purpose AI into a highly reliable space engineer. It's like taking a genius student and giving them a perfect, structured internship rather than just handing them a stack of textbooks.

Technical Summary

Here is a detailed technical summary of the paper "Cognitively Layered Data Synthesis for Domain Adaptation of LLMs to Space Situational Awareness."

1. Problem Statement

The paper addresses the challenge of adapting general-purpose Large Language Models (LLMs) to complex, high-stakes engineering domains, specifically Space Situational Awareness (SSA). While LLMs excel in general tasks, they struggle in SSA due to three critical gaps in existing Supervised Fine-Tuning (SFT) data construction methods:

• Inadequate Structured Coverage: Existing corpora lack organization aligned with the SSA mission chain (detection, tracking, prediction, assessment, disposition), leading to systematic knowledge gaps.
• Shallow Cognitive Depth: Public datasets are dominated by factual recall and conceptual paraphrasing, lacking verifiable supervision for higher-order cognitive tasks (analysis, evaluation, decision-making) required for engineering workflows.
• Misalignment with Engineering Specifications: General quality assessment methods fail to account for engineering constraints, technical rigor, and workflow compliance, making it difficult to scale data while ensuring domain reliability.

The core bottleneck identified is the construction of high-quality, structured SFT datasets that bridge the gap between general linguistic competence and executable, verifiable engineering reasoning.

2. Methodology: The BD-FDG Framework

The authors propose BD-FDG (Bloom's Taxonomy-based Domain-specific Fine-tuning Data Generation), a three-stage framework designed to synthesize high-quality SFT data.

A. Domain Knowledge Base Construction (Mission-Chain Driven)

• Source: Foundational design literature and technical documents regarding SSA missions.
• Structure: A hierarchical Knowledge Tree is constructed, organizing data into three tiers: System-level tasks $\rightarrow$ Subsystem modules $\rightarrow$ Key technical units.
• Processing: Documents are parsed using MinerU (a vision-language tool) to extract structured text, tables, and formulas.
• Retrieval: A Hybrid Retrieval system is built using dense embeddings (text-embedding-v3) and sparse indexing (BM25). For any query, the system retrieves and reranks top-$K$ chunks using a hybrid score ($Score_{hybrid} = \alpha s_{dense} + (1-\alpha) s_{BM25}$) to ensure both semantic relevance and keyword precision.

B. Cognitively Layered Question Generation

• Theoretical Basis: The framework operationalizes Bloom's Taxonomy (Remember, Understand, Apply, Analyze, Evaluate, Create).
• Question Types: Nine domain-specific question types (Q1–Q9) are defined, mapping the six cognitive levels to SSA tasks (e.g., Concept Discrimination, Algorithm Implementation, Solution Decision-Making).
• Generation Process:
  1. A "Teacher" LLM (QwQ-Plus) receives a multisource context window (retrieved chunks).
  2. Guided by Bloom-based prompt templates, it generates questions, reasoning traces (Chain-of-Thought), and answers.
  3. The process creates a continuous difficulty gradient from basic recall to complex system design.

C. SFT Data Synthesis & Quality Control

• Distillation: Each filtered question is distilled 16 times (X16) to generate diverse reasoning paths and mitigate single-path bias, expanding the dataset scale.
• Quality Filtering: A "Student" LLM (Qwen-Max) acts as a filter, scoring samples based on a four-dimensional rubric:
  1. Domain-Specific Evaluation: Technical soundness and alignment with SSA workflows.
  2. Self-Containment Evaluation: Completeness without reliance on omitted context.
  3. Structured Scoring: Assessment of logical coherence and consistency.
  4. Key Deduction/Bonus: Penalties for factual errors; bonuses for rigorous reasoning.
• Output: The final dataset, SSA-SFT, contains ~230K samples, with ~60% dedicated to higher-order cognitive tasks (Q5–Q9).

3. Key Contributions

1. BD-FDG Framework: A novel, scalable pipeline that couples mission-chain-driven knowledge organization with Bloom's Taxonomy-guided cognitive layering to solve the "shallow depth" and "unstructured coverage" problems in domain adaptation.
2. SSA-SFT Dataset: The creation of a large-scale, high-quality dataset (~230K samples) specifically for Space Situational Awareness, featuring a balanced distribution across nine question types and six cognitive levels.
3. Engineering-Aligned Quality Control: A multidimensional scoring pipeline that explicitly enforces engineering constraints and workflow compliance, moving beyond generic text quality metrics.
4. Empirical Validation: Demonstration that cognitively structured supervision signals significantly improve domain performance while preserving general capabilities.

4. Experimental Results

The authors fine-tuned Qwen3-8B using SSA-SFT to create SSA-LLM-8B and evaluated it against the baseline on the SSA-Test set (1,644 samples) and general benchmarks.

• Domain Performance (SSA-Test):

  • BLEU-1 Improvement: Massive relative gains of 144% (no-think mode) and 176% (think mode) over the baseline.
  • Arena Battle: SSA-LLM-8B achieved an 82.21% win rate (no-think) and 73.54% (think) against the baseline, judged on professionalism, completeness, and usability.
  • Reasoning Impact: The "think" mode (Chain-of-Thought) provided a significant boost (+9.9% BLEU-1) for the fine-tuned model, indicating that explicit reasoning complements internalized domain knowledge.

• General Capability Retention:

  • Mathematics: Performance remained stable or slightly improved (e.g., MATH-500 scores ranged from 84.60% to 94.80%).
  • Knowledge/Exams: Minor trade-offs observed (e.g., slight declines in C-Eval and GPQA-Diamond), but core benchmarks like MMLU-Pro and MMLU-Redux remained largely competitive.
  • Instruction/Code: Modest declines in instruction following (IFEval) and code generation (LiveCodeBench), attributed to the domain-heavy nature of the training data.

• Hyperparameter Analysis:

  • Optimal retrieval configuration was found at $\alpha=0.50$ (balanced semantic/keyword weight) and $K=5$ (Top-5 chunks), achieving a quality score of 7.55. This highlights an "inverted-U" effect where too little context starves the model, and too much introduces noise.

5. Significance and Conclusion

This paper establishes a new paradigm for domain adaptation in complex engineering fields. It demonstrates that:

• Cognitive Layering is Crucial: Simply increasing data volume is insufficient; data must be structured to span a continuous difficulty gradient from recall to creation.
• Mission-Chain Alignment: Organizing knowledge around the actual engineering workflow (detection $\rightarrow$ disposition) is essential for generating verifiable, executable reasoning.
• Scalability: The BD-FDG framework successfully scales the construction of high-quality, specialized datasets (230K+ samples) using automated distillation and rigorous filtering, offering a transferable solution for other high-stakes domains like autonomous driving or power grid operations.

The work concludes that coupling cognitive frameworks with structured domain knowledge and engineering-aligned quality control effectively bridges the gap between general LLMs and specialized engineering requirements.