RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

1. Problem Statement

Fine-tuning Large Language Models (LLMs) for specific domains is often computationally expensive. While Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) mitigate this by learning low-rank updates, current implementations typically apply adaptation uniformly across all layers.

This uniform approach ignores a fundamental property of deep networks: different layers play distinct geometric and functional roles. Existing sparse adaptation strategies often rely on heuristics, randomization, or module fusion without a principled foundation in the model's intrinsic structure. Consequently, there is a lack of a robust, training-free signal to determine which specific layers are most critical for adaptation, leading to suboptimal parameter efficiency.

2. Methodology: RDP LoRA

The authors propose a geometry-driven framework that treats the sequence of hidden states in a transformer model as a high-dimensional geometric trajectory. The core idea is to identify "structural pivot points" along this trajectory where significant semantic transformations occur, and to apply LoRA adaptation exclusively to these layers.

Core Components:

• Trajectory Modeling: The sequence of hidden states $V = \{v_1, v_2, ..., v_L\}$ across $L$ layers is modeled as a discrete geometric curve in a high-dimensional embedding space ($\mathbb{R}^D$).
• Layer Representation: To summarize the semantic state of each layer, the authors use an Attention-Weighted Projection. Instead of simple mean pooling or using the last token, they compute a weighted sum of hidden states based on the attention weights of the final token, capturing the model's contextual focus at each depth.
• The RDP Algorithm: The Ramer–Douglas–Peucker (RDP) algorithm, a classical curve simplification method from cartography, is applied to these trajectories.
  • Mechanism: RDP iteratively identifies points with the maximum orthogonal distance from a reference line connecting trajectory endpoints. If this deviation exceeds a threshold, the point is marked as a "structural pivot."
  • Dimension-Agnostic: Because RDP relies solely on Euclidean distance, it works effectively in high-dimensional embedding spaces (e.g., 768+ dimensions) without modification.
  • Function: It filters out local "noise" (redundant micro-variations) while preserving the global structural skeleton (significant semantic shifts).
• Multi-Scale Analysis & Voting:
  • Instead of a fixed threshold, the authors use a target-driven approach. They run RDP across a range of target resolutions ($t$), automatically computing the necessary threshold $\epsilon_t$ to retain $t$ points.
  • Importance Scoring: Layers selected as pivots across multiple resolutions are aggregated. A weighted voting score ($\omega_{RDP}$) is calculated, giving higher weight to pivots that persist at coarser resolutions (global structure) while retaining sensitivity to finer scales.
• Hybrid Signal: The final layer importance score combines the RDP structural score with a velocity-aware measure (rate of change between layers) to identify the "Reasoning-Relevant Band."

Adaptation Strategy:

Once critical layers are identified, the model is frozen except for these selected "pivot" layers. LoRA adapters are applied only to these layers, significantly reducing the number of trainable parameters.

3. Key Contributions

1. Training-Free Layer Selection: Introduces a method to select layers for PEFT based entirely on the intrinsic geometry of the model's forward pass, requiring no gradient computation or additional training data for the selection process.
2. Geometric Interpretation of Semantics: Demonstrates that semantic transitions in LLMs correspond to high-curvature turns in the embedding trajectory, which can be isolated using curve simplification algorithms.
3. Superior Sparse Adaptation: Proves that selecting layers based on geometric importance yields better performance than random selection or uniform adaptation, even with far fewer parameters.
4. Dimension-Agnostic Framework: Extends a 2D/3D geometric algorithm (RDP) to high-dimensional embedding spaces, providing a principled way to analyze deep network hierarchies.

4. Experimental Results

The method was evaluated on Qwen3-8B-Base using the MMLU-Math benchmark, with comparisons against other models (Qwen3-4B/14B, DeepSeek-LLM-7B, Gemma-7B).

• Performance:
  • Baseline (No Adaptation): 74.25%
  • Full LoRA (All 36 layers): 79.32%
  • Random Sparse LoRA (13 random layers): 75.56%
  • Reasoning-Band LoRA (All layers in the identified band): 78.10%
  • RDP LoRA (Top 13 geometrically selected layers): 81.67%
• Key Findings:
  • Adapting only 13 layers (approx. 36% of the model) selected via RDP outperformed adapting all 36 layers by over 2 percentage points.
  • The performance gap between RDP selection and random selection (6.11%) highlights that layer identity is more critical than the sheer number of adapted parameters.
  • The method is robust across different model scales and architectures, consistently outperforming random baselines and matching or exceeding full LoRA performance with significantly fewer parameters.

5. Significance and Implications

• Efficiency vs. Performance: The paper challenges the assumption that more parameters (or more layers) equal better performance. It shows that structural location is the primary driver of adaptation success.
• Interpretability: The method provides an interpretable signal for why certain layers are chosen (they represent major semantic shifts), moving beyond "black box" heuristic tuning.
• Scalability: By reducing the number of trainable parameters while increasing performance, RDP LoRA offers a highly scalable solution for adapting large models to specialized domains without the computational cost of full fine-tuning or uniform LoRA.
• Future Directions: The authors suggest extending this static geometric signal to dynamic, input-dependent layer selection during inference or online training updates.

In conclusion, RDP LoRA establishes that the intrinsic geometry of hidden-state trajectories contains a robust, training-free signal for optimizing parameter-efficient fine-tuning, allowing models to achieve superior reasoning capabilities with a fraction of the trainable parameters.