💻 computer science

A Controlled Candidate-Set Benchmark for Offline Satellite-Security Plan Decomposition

This paper introduces a controlled candidate-set benchmark and a low-rank decomposition adapter for offline satellite-security plan decomposition, demonstrating that while the adapter improves format compliance and selection precision over prompting at specific model sizes, it does not yet constitute a reliable standalone system due to significant training variance and low autoregressive accuracy.

Original authors: João Paolo Cavalcante Martins Oliveira, Lucas Teske, Paulo Matias

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: João Paolo Cavalcante Martins Oliveira, Lucas Teske, Paulo Matias

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship, but instead of steering with a joystick, you have to give the ship a written list of instructions to fix a broken engine. The problem is, you can't just write "fix engine"; you need a specific, step-by-step plan using only the tools you have in your toolbox. This is the world of satellite security, where experts need to map out exactly how a hacker might try to break into a satellite, so they can build better defenses. To do this, they use Large Language Models (LLMs), which are like super-smart robots that read and write text. Usually, these robots are huge and need massive computers to run. But what if you could shrink that brain down to fit on a smaller, local computer that stays safe inside a secure room? That's the big question: Can a tiny, local robot be smart enough to figure out a complex security plan without accidentally leaking secrets or making up fake steps?

This paper is like a rigorous test drive for a new kind of "training wheels" called a low-rank adapter. Think of the main robot brain (the LLM) as a giant, frozen library that knows everything but is too heavy to move. The adapter is a small, lightweight backpack you strap onto it. This backpack is trained to help the robot pick the right tools from a specific list and put them in the right order. The researchers built a special "training gym" with 24 different satellite security scenarios. They didn't just ask the robot to guess; they gave it a shuffled deck of cards containing the correct moves mixed with some wrong ones, and asked it to pull out only the right cards and lay them out in the correct sequence.

The results are a bit like a mixed bag of "good effort, but not ready for the big leagues." When they tested the robot with the backpack on the 1.5-billion-parameter model (a medium-sized brain), it got about 58% of the right moves and picked them out with 58% accuracy. That sounds okay, but when they compared it to a simpler method where they just gave the robot two examples of how to do it (called "two-shot prompting"), the simpler method actually found more of the right moves (66% recall), even if it was a bit messier about picking the wrong ones. The adapter was much better at following the rules of how to write the answer, rarely messing up the format. However, the authors emphasize that this high format compliance is a confounding factor; because the adapter followed the structure so much better, we cannot simply attribute all the score differences to the adapter being better at selecting the right security steps. The robot did stumble hard when trying to figure out just the next step in a long chain of events without seeing the whole plan, getting the right next move less than 30% of the time even on the largest model.

The authors are very careful not to call this a "win." They explicitly state that this isn't a magic bullet that solves satellite security on its own. In fact, they rule out the idea that this adapter is a general upgrade for all small models. The study shows that while the adapter helps the robot follow the format and pick the right tools from a controlled list, it doesn't necessarily make the robot smarter at planning the whole mission from scratch. The "training wheels" work well for keeping the robot on the path and following instructions, but they don't guarantee the robot will always know the best path to take. The paper concludes that this is a useful "proof of concept"—a way to test if a robot can pick the right tools from a list without leaking secrets—but it's not yet a reliable, stand-alone system for real-world defense. The researchers are offering their data and tools as a starting point for others to keep improving, rather than a finished product ready for deployment.

Technical Summary: A Controlled Candidate-Set Benchmark for Offline Satellite-Security Plan Decomposition

Problem Formulation

The paper addresses a specific component of satellite security development: mapping a high-level security objective (e.g., testing telecommand authorization) to an ordered, checkable plan of action. In environments where sensitive artifacts cannot leave a local setting, compact local models are preferred. However, existing research on decomposition prompting and task specialization has not established whether these strategies work for security-campaign plans.

The core challenge is defined as a candidate-set objective decomposition task. Given an objective $o$ and a set of available techniques $G(o)$ , the model must select the correct reference techniques and order them into a sequence $\sigma(o)$ .

The Reference: A decomposition is an ordered sequence of triples $(t_i, \tau_i, c_i)$ , where $t_i$ is a natural language action, $\tau_i$ is a technique identifier from the SPARTA framework, and $c_i$ is an authored check statement.
The Constraint: The model operates within a "controlled candidate set" $G(o)$ . In this study, $G(o)$ is an oracle set constructed to include every reference technique plus in-domain distractors (techniques from different families). The task is to select and order the correct subset, not to retrieve techniques from a vast corpus or generate them from scratch.
The Goal: To measure selection (precision/selectivity) and ordering fidelity, distinguishing the model's ability to pick the right tools from the ability to simply dump the entire candidate set (which would yield high recall but low precision).

Methodology

The authors propose a low-rank adapter (LoRA) trained to emit a plan conditioned on an explicit candidate set, separating the decomposition component from retrieval and verification layers.

Dataset Construction (SatSec Corpus):
- Source: 24 authored cases grounded in SPARTA and space-security standards, covering incidents like GNSS spoofing, passive eavesdropping, and unauthenticated telecommand analysis.
- Composition: 24 author-written "decompose" examples and 83 mechanically derived "next-step" examples, totaling 107 examples.
- Split: The dataset is split case-disjointly. Six cases are held out for evaluation, ensuring no steps from a test case appear in the training set.
- Leakage Control: To prevent reference leakage, input prompts contain only the objective and eight shuffled candidates (the reference techniques plus distractors). The reference narrative and step sequences are removed from inputs.
- Distractors: Candidates are padded to eight distinct techniques. Distractors are drawn from different technique families to ensure the task is cross-family selection, not sibling-level discrimination.
Model Architecture & Training:
- Base Models: Qwen2.5-0.5B, 1.5B, and 7B-Instruct.
- Adapter: A low-rank adapter (Rank 16, $\alpha=32$ ) is trained on the frozen base model.
- Supervision: Two modes are used:
  - Full Decomposition: Autoregressive generation of the entire plan.
  - Next-Step: Supervising a single continuation given previous steps.
- Loss Function: Completion-only masking is used, meaning the loss is calculated only on the generated tokens, not the prompt (which includes the candidate set).
Evaluation Protocol:
- Baselines: Unadapted models with candidates only, schema-prompted models, and two-shot prompting with fixed exemplars.
- Metrics:
  - Recall: Fraction of reference techniques recovered.
  - Precision (Selectivity): Fraction of emitted techniques that are in the reference set (penalizing distractor inclusion).
  - Ordering Fidelity: Precedence violations among matched steps.
  - Candidate Validity: Whether emitted identifiers are valid SPARTA entries from $G(o)$ .
  - Check-Field Presence: Syntactic presence of the check field.
- Stability Checks: Evaluation across five decoding seeds and three training seeds (42, 43, 44) to distinguish sampling noise from training variance. A 24-fold leave-one-case-out (LOCO) census is performed at 0.5B and 1.5B.

Key Results

The evaluation reveals nuanced trade-offs rather than a clear superiority of the adapter approach across all metrics and model sizes.

Precision vs. Recall Trade-off:
- At 1.5B, the adapter achieves higher precision (0.583) than two-shot prompting (0.480) but lower recall (0.586 vs. 0.660).
- At 7B, the adapter shows higher precision (0.67) than schema prompting (0.59) but lower ordering fidelity (0.68 vs. 0.84).
- At 0.5B, the adapter slightly outperforms two-shot prompting in precision (0.27 vs. 0.20) but lags in recall.
Format Compliance: The adapter is substantially more format-compliant. At 1.5B, 96.7% of adapter predictions contain complete structured fields, compared to 30.0% for schema prompting. This suggests a significant portion of the score difference is due to syntactic adherence rather than semantic selection quality.
Next-Step Accuracy: Autoregressive next-step accuracy is low, particularly for smaller models (0.08 at 0.5B, 0.06 at 1.5B, 0.29 at 7B).
Stability: Training variance is non-negligible. Case-resampling intervals for precision relative to the strongest baseline cross zero at 1.5B and 7B, indicating that the observed gains are not statistically robust across all cases.
Transfer Diagnostic: A separate test on three public crackmes showed that while the concept of the action-technique-check contract can be reused (with post-hoc normalization), the exact contract rate was low (0.133), and the adapter itself was not evaluated in this transfer scenario.

Significance and Claims

The authors explicitly frame this work as a controlled proof of concept, not evidence of a reliable stand-alone decomposition system or a general adapter advantage.

Contribution: The paper provides a formulation for candidate-set decomposition, a leakage-controlled dataset (SatSec), and a reproducibility package including raw generations and audit tools.
Scope Limitations: The study isolates the decomposition front-end. It does not evaluate retrieval (since oracle sets are used), semantic verification of the checks, or generalization to new techniques without retraining.
Modest Conclusions:
- The results show a precision-recall trade-off rather than a universal improvement.
- The observed performance gains are partially confounded by format learning (the adapter learns to output structured JSON better than baselines).
- The small corpus and case-disjoint split characterize these specific 24 cases but do not estimate performance on a broader population.
- The low next-step accuracy indicates the system is not yet ready for autonomous, long-horizon planning.

The paper concludes that the artifact serves as an auditable starting point for testing candidate selection separately from retrieval and verification. It calls for future work involving community semantic review, genuinely untouched external test sets, realistic retrieval integration, and on-device profiling before any claims of reliability or deployment readiness can be made.

Technical Summary: A Controlled Candidate-Set Benchmark for Offline Satellite-Security Plan Decomposition

Problem Formulation

Methodology

Key Results

Significance and Claims

More like this