Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency

1. Problem Statement

Large Language Models (LLMs) increasingly operate in complex environments where they must process multiple instructions from heterogeneous sources with varying authority levels (e.g., system policies, user requests, tool outputs, and retrieved context). While prior research has focused on adversarial attacks (e.g., prompt injection, jailbreaking) to prevent security violations, it has largely overlooked benign but common instruction conflicts that arise in real-world applications.

Key Challenges:

• Instruction Hierarchy: Models must respect a strict priority order (System > User > Tools/History).
• Benign Conflicts: Real-world scenarios often involve non-malicious but contradictory instructions (e.g., a system requiring JSON output vs. a user requesting plain text).
• Current Limitations: Existing models frequently fail to resolve these conflicts correctly, either violating high-priority system constraints or losing task utility when lower-priority instructions contradict higher ones.
• Gap: There is a lack of methods that ensure models can resolve these conflicts while maintaining helpfulness and behavioral consistency without relying solely on adversarial training.

2. Methodology: Neuro-Symbolic Hierarchical Alignment (NSHA)

The authors propose NSHA, a framework that combines neural language models with symbolic reasoning to enforce hierarchical instruction following. The approach consists of two main phases:

A. Inference-Time: Solver-Guided Reasoning

At inference time, the system treats instruction resolution as a Constraint Satisfaction Problem (CSP) solved via a Satisfiability Modulo Theories (SMT) solver (specifically Z3). The pipeline involves:

1. Atomization: A parser decomposes the raw context (system, user, tool messages) into discrete atomic instructions, each annotated with an authority level ($0$ for System, $1$ for User, $2$ for Tools/Docs).
2. Neural Conflict Scanning: A pre-trained Natural Language Inference (NLI) model (cross-encoder) evaluates pairs of atomic instructions to detect semantic contradictions. A conflict matrix is generated where $M_{ij}=1$ if instructions $I_i$ and $I_j$ are contradictory.
3. Hierarchical Optimization (MaxSMT): The problem is formulated as a lexicographic MaxSMT optimization:
   • Hard Constraints: No two contradictory instructions can be active simultaneously ($\neg z_i \lor \neg z_j$).
   • Lexicographic Weights: Instructions are assigned weights based on authority ($w_i = B^{K-A(I_i)}$). The solver maximizes the weighted sum of selected instructions, ensuring higher-authority instructions are preserved unless they conflict with even higher ones.
   • Output: The solver produces a maximally consistent subset of instructions ($A^*$) that respects the hierarchy.
4. Guided Generation: The LLM generates a response using only the refined prompt containing the selected instructions from $A^*$.

B. Training-Time: Hierarchical Alignment

To eliminate the latency and overhead of external solvers during deployment, the authors distill the solver's logic into the model parameters:

1. Dataset Construction: Using the Alpaca corpus, they synthesize training data containing:
   • Aligned contexts: Compatible instructions.
   • Conflict contexts: Intentional contradictions between authority levels.
   • Labels: The solver determines the "accepted" (hierarchy-compliant) and "rejected" (hierarchy-violating) responses for each context.
2. Hierarchy-Consistent Alignment Loss (HCAL): The model is fine-tuned using a composite loss function:
   $$L_{HCAL} = L_{pref} + \gamma \cdot L_{SL} + \beta \cdot L_{KL}$$
   • $L_{pref}$: Standard preference loss (DPO-style) to rank accepted responses higher than rejected ones.
   • $L_{SL}$: Semantic Loss, which explicitly penalizes the probability mass assigned to hierarchy-violating outcomes, forcing the model to concentrate probability on valid constraint satisfaction.
   • $L_{KL}$: Regularization to preserve general capabilities.

3. Key Contributions

• Novel Problem Formulation: Shifts focus from adversarial prompt injection to benign instruction conflicts in multi-source settings, formalizing instruction hierarchy as a logical constraint satisfaction problem.
• Neuro-Symbolic Framework (NSHA): Introduces a dual-phase approach combining solver-guided inference (for verifiable, principled resolution) and training-time distillation (for efficient deployment).
• Automated Supervision: Develops a method to automatically construct high-quality training data with solver-derived ground truth for hierarchy compliance.
• Comprehensive Evaluation: Evaluates across four domains: Rule Following, Task Execution, Tool Use, and Safety, covering both single-turn and multi-turn interactions.

4. Experimental Results

The method was evaluated on Qwen3-4B-Instruct and Llama-3.1-8B-Instruct across the IHEval benchmark.

• Rule Following:
  • Baseline models suffer drastic performance drops in conflict scenarios (e.g., Qwen drops from ~89% to ~26% in multi-turn conflicts).
  • NSHA-DPO achieved the best results, significantly improving conflict resolution (e.g., raising Qwen conflict scores to ~36-47%) while maintaining high performance in non-conflicting settings.
• Task Execution:
  • Conflicts severely degrade utility (e.g., language detection accuracy dropping to near 0%).
  • NSHA variants (particularly DPO and HCAL) successfully recovered performance, demonstrating that hierarchy awareness preserves task utility even under conflicting constraints.
• Tool Use:
  • Neuro-symbolic inference (NS) showed strong robustness by decoupling interpretation from execution.
  • NSHA-HCAL achieved the best performance on Llama models, outperforming baselines in tool invocation under conflict.
• Safety:
  • Hierarchy-aware alignment improved robustness against prompt hijacking and extraction attacks.
  • NSHA-DPO maintained high reference scores while achieving the best safety scores in conflict scenarios for Llama models.
• Multi-Turn Robustness:
  • Analysis revealed that models often fail to maintain hierarchy over multiple turns (long-horizon drift).
  • NSHA-DPO demonstrated superior stability in multi-turn settings, effectively preserving system priorities across repeated conflicts.

5. Significance

• Real-World Applicability: Addresses the critical gap between theoretical safety and practical utility in agentic systems where benign conflicts are frequent.
• Verifiability: The neuro-symbolic approach provides a transparent, logical mechanism for conflict resolution, reducing "black box" decision-making.
• Scalability: By distilling solver logic into model weights, the approach enables efficient deployment without requiring external solvers at inference time.
• Future Agentic Systems: The work lays the groundwork for reliable long-horizon agents that must continuously reconcile system policies, user goals, and tool outputs without compromising safety or utility.

In conclusion, NSHA offers a robust solution for hierarchical instruction following, proving that integrating symbolic logic with neural training significantly enhances an LLM's ability to navigate complex, multi-instruction environments.