💬 NLP

SyRuP: Enhancing System-Prompt Following via Reward-Guided Prediction in LLM Decoding

The paper introduces SyRuP, a decoding-time framework that enhances Large Language Model adherence to system prompts by training a cross-attention reward head to generate token-level adherence scores for reranking candidates without requiring base model tuning.

Original authors: Seoyeon Kim, Minjae Kang, Jaehyung Kim

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seoyeon Kim, Minjae Kang, Jaehyung Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, well-read robot that has read almost every book in the library. This robot is a Large Language Model (LLM). Usually, when you ask it a question, it answers based on its training. But sometimes, you want to give it a specific set of rules before it starts talking. You might say, "Act like a grumpy pirate," or "Only answer in rhymes," or "Never mention violence." These rules are called system prompts. They are like the job description or the costume the robot wears for a specific conversation.

The problem is that these robots are used to following instructions that appear inside the conversation, not rules set in stone before the chat begins. When the rules get complicated—like "be a pirate, but also use 19th-century grammar, avoid the letter 'e', and keep it funny"—the robot often forgets the rules halfway through. It might start talking like a pirate, then suddenly switch to a modern teenager, or forget to avoid the letter 'e'. Scientists have tried to fix this by retraining the robot (which is expensive and slow) or by trying to guess the best answer after it's already written (which is like editing a novel after the author has already finished it). The big question is: Can we guide the robot to follow these complex rules while it is writing, without changing its brain or waiting until the end?

This is where a new method called SYRUP comes in. Think of the robot's brain as a massive, frozen library that we aren't allowed to rearrange. SYRUP is like a super-smart, tiny librarian who stands right next to the robot while it writes. Every time the robot picks a word to say next, this librarian checks two things: "Does this word fit the story?" and "Does this word fit the pirate costume?"

Here is how it works in practice. The robot has a list of its top 10 favorite words to say next. Usually, it just picks the number one favorite. But SYRUP intervenes. It looks at the "system prompt" (the pirate costume) as a separate, special memory card. It uses a special tool called a cross-attention reward head to ask the robot's current thoughts, "Hey, if you say this word, does it match the pirate vibe?" The librarian then gives a score to each of the top 10 words. If the robot wants to say "Hello," but the pirate costume requires "Ahoy," the librarian boosts the score for "Ahoy" and lowers the score for "Hello." The robot then picks the word with the highest combined score.

The paper suggests that this approach is much better than the old ways. The researchers tested SYRUP on three different robot brains and found that it consistently helped them follow complex rules better than just giving the robot a better prompt or trying to re-rank answers after they were written. In fact, on one test, SYRUP improved the robot's performance by about 5.6% compared to the next-best method. That might sound small, but in the world of AI, that's a huge jump.

The paper also rules out a few other ideas. It shows that simply merging the rules into the main conversation text (like hiding the pirate instructions inside the user's question) isn't enough; the robot needs the rules to be treated as a separate, distinct memory to really follow them. It also found that while you can make the robot follow rules better by retraining it from scratch, that takes too much time and money. SYRUP is a "decoding-time" method, meaning it works while the robot is thinking, keeping the original brain frozen and only training a tiny, lightweight add-on.

Interestingly, the researchers found that SYRUP works even on rules it hasn't seen before. They trained it on one set of complex instructions and then tested it on a completely different set of rules (like strict formatting or word counts), and it still performed well. This suggests the method teaches the robot a general skill for "listening to the boss" rather than just memorizing specific answers.

However, the paper is careful to note that this isn't a magic wand that solves everything. The method does add a tiny bit of extra time to the robot's thinking process because the librarian has to check the words. But the authors suggest this cost is very small compared to other complex methods that try to do the same thing. They also warn that if you push the "rule-following" too hard, the robot might start sounding weird or unnatural, so there has to be a balance.

In short, SYRUP suggests that if you want a robot to follow a complex set of rules without retraining its entire brain, you shouldn't just hope it remembers. Instead, you should give it a dedicated, real-time guide that checks every single word it wants to say against the rules, ensuring the final story stays true to the costume it was wearing from the very beginning.

Technical Summary: SyRuP – Enhancing System-Prompt Following via Reward-Guided Prediction

Problem Statement

Large Language Models (LLMs) are increasingly controlled via system prompts that define roles, styles, formats, and safety constraints. However, current models follow these prompts implicitly through in-context learning, which often proves insufficient for complex or compositional instructions. Existing solutions face significant trade-offs:

Training-time methods (e.g., SFT, RLHF) require costly parameter updates and may incur an "alignment tax," degrading general capabilities.
Decoding-time methods often rely on implicit instruction-following or treat the system prompt merely as part of a merged context sequence. Consequently, the system prompt loses its distinct role as a separate conditioning source, and prior reward-guided approaches have focused on fixed attributes (e.g., toxicity, sentiment) rather than instance-specific system-prompt adherence.

There is a lack of frameworks that explicitly treat the system prompt as a separate memory source to provide token-level guidance during inference without modifying the base model.

Methodology: SyRuP

The authors propose SyRuP (System-prompt following via Reward-guided Prediction), a decoding-time framework that keeps the base LLM frozen and trains only a lightweight reward head. The core innovation is reformulating system-prompt following as a token-level conditional adherence problem.

1. Cross-Attention Reward Head

Unlike standard reward models that merge the system prompt, user input, and generated prefix into a single sequence, SyRuP maintains the system prompt as a distinct Key-Value (KV) memory.

Architecture: The reward head uses a cross-attention mechanism where the hidden states of the system prompt ( $H(s)$ ) serve as keys ( $K_s$ ) and values ( $V_s$ ).
Query Construction: The query ( $Q_t$ ) is derived from the concatenation of the user input and the generated prefix ( $H(x), H(y_{<t})$ ).
Scoring: The cross-attention output attends to the most relevant parts of the system prompt for the current generation state. This is fused with the context representation and passed through a two-layer MLP to produce a token-level adherence score ( $r_t(v)$ ) for each candidate token.
Training: The head is trained on system-prompt-conditioned preference pairs $(s, x, y^+, y^-)$ using a Bradley-Terry objective. The base LM remains frozen; only the cross-attention projections and the fusion MLP are updated.

2. Reward-Guided Decoding

At inference, SyRuP reranks the top- $k$ candidates from the base LM's logits. The final selection score combines three components:
$\text{Score}(v) = \ell_t(v) + \lambda_r r_t(v) + \lambda_c \Delta_t(v)$

$\ell_t(v)$ : Base LM logits (fluency).
$r_t(v)$ : Learned token-level reward score (explicit adherence).
$\Delta_t(v)$ : An optional contrastive signal measuring the logit shift induced by the system prompt relative to a neutral prompt ( $\ell_t(v) - \ell^{neu}_t(v)$ ).
$\lambda_r, \lambda_c$ : Hyperparameters controlling the strength of the reward and contrastive signals.

Key Contributions

Explicit System-Conditioning: SyRuP is the first decoding-time framework to treat the system prompt as a separate memory source via cross-attention, enabling explicit token-level adherence evaluation rather than relying on implicit context absorption.
Lightweight Inference-Time Control: The method requires no updates to the base LLM parameters, only a small, trainable reward head, offering a practical alternative to full fine-tuning.
Hybrid Guidance: The introduction of a hybrid variant that combines explicit reward guidance with a contrastive signal to capture system-induced distribution shifts, providing complementary signals for steering generation.

Experimental Results

The authors evaluated SyRuP on Multifaceted-Bench (in-domain) and IFEval (out-of-distribution) using three backbone models (Qwen-2.5-1.5B, Llama-3.2-3B, Llama-3.1-8B).

Performance: SyRuP consistently outperformed all baselines, including prompting methods (e.g., Few-shot, Marked Prompting) and decoding-time baselines (e.g., OPAD, GenARM, ARGS, BoN, DeAL).
- On Multifaceted-Bench, it achieved the best average score across all 15 model-task combinations, improving the strongest baseline by 5.61% on Qwen-2.5-1.5B.
- On IFEval, without any task-specific training, SyRuP achieved the best average strict prompt-level accuracy, demonstrating strong generalization to verifiable constraints.
Efficiency: SyRuP offers a favorable trade-off between adherence and cost.
- The reward-only variant increased latency by only 1.04× compared to the base model.
- The full hybrid variant incurred a 2.18× latency increase, which remains significantly lower than other decoding-time baselines like ARGS (5.13×) and DeAL (16.83×).
- Peak GPU memory usage remained close to the base model (3.1 GB vs. >10 GB for some baselines).
Analysis: Ablation studies confirmed that explicit cross-attention conditioning outperforms single-stream implicit conditioning. Furthermore, the reward and contrastive signals were found to be complementary, with the combined approach yielding unique token choices not selected by either signal alone.

Significance and Claims

The paper claims that SyRuP validates the hypothesis that explicit token-level guidance is an effective and practical mechanism for reliable system-prompt following. By decoupling the system prompt from the generation context and treating it as a dedicated memory source, SyRuP enables models to adhere to complex, diverse, and instance-specific constraints without the computational overhead of retraining or the high inference costs of search-based decoding.

The authors position SyRuP as a step toward more controllable LLM generation, particularly for scenarios where system prompts define critical roles, safety guidelines, or formatting requirements that standard in-context learning fails to enforce consistently. They acknowledge limitations, such as the additional overhead of the hybrid contrastive signal and the potential for over-steering, suggesting future work in adaptive weighting and safety filtering.