Technical Summary: Towards Building Non-Fine-Tunable Foundation Models

1. Problem Statement

The open-sourcing of Foundation Models (FMs), particularly Large Language Models (LLMs), facilitates broad reuse and scientific innovation but exposes model trainers to significant economic and safety risks. Once weights are released, users can perform low-cost, unauthorized fine-tuning on downstream tasks. This creates two primary issues:

• Economic Risk: Users can capture value from downstream adaptations that would otherwise belong to the original trainer.
• Safety Risk: Trainers lose control over the model's lifecycle. Unauthorized fine-tuning on harmful data (e.g., violence, extremism, discrimination) can generate harmful outputs while retaining the reputation of the original pre-trained model, potentially leading to legal and regulatory liabilities.

Existing defenses, such as model immunization or task-specific unlearning, rely on knowing the specific downstream tasks or data distributions to be restricted. These approaches are impractical for general open-weight releases where trainers lack visibility into how models will be repurposed. The paper addresses the need for a mechanism that allows open-source release while intrinsically limiting the effectiveness of unauthorized, task-agnostic fine-tuning.

2. Methodology: Private Mask Pre-Training (PMP)

The authors propose Private Mask Pre-Training (PMP), a pre-training paradigm designed to make non-fine-tunability an intrinsic property of the model's geometry. The core idea is to concentrate representation learning into a sparse subnetwork identified early in training, while keeping the defining binary mask private.

The PMP Procedure

PMP operates in three stages:

1. Private Early-Bird Mask Discovery:

   • The model undergoes a short "warm-up" pre-training phase on a small subset of data.
   • Using an "early-bird lottery ticket" criterion, the method identifies a stable, sparse binary mask ($M$) based on the magnitude of gradients during this phase.
   • The mask is selected when consecutive masks stabilize (measured by Intersection over Union, IoU).
   • This mask $M$ is kept private and fixed for all subsequent training.

2. Private Mask Pre-Training:

   • The model is trained on the full pre-training dataset, but optimization is strictly restricted to the parameters defined by the mask ($\theta_M$).
   • The complement parameters ($\theta_{\bar{M}}$) remain frozen throughout the entire pre-training process.
   • Although the forward pass utilizes the full dense parameter vector, all gradient updates are confined to the sparse subnetwork.

3. Release Without Mask:

   • Upon convergence, the trainer releases only the final dense weights.
   • The private mask $M$ is not disclosed.

Mechanism of Defense

When an unauthorized user attempts to fine-tune the released model without the mask, they must update the full parameter vector. This creates an intrinsic optimization mismatch:

• Pre-training: Optimized a conditional objective where updates were restricted to $\theta_M$, while $\theta_{\bar{M}}$ remained fixed.
• Unauthorized Fine-tuning: Optimizes a joint objective updating both $\theta_M$ and $\theta_{\bar{M}}$.
• Result: The fine-tuning process forces updates on the frozen complement parameters ($\theta_{\bar{M}}$), which were never adapted during pre-training. Theoretical analysis suggests the loss landscape along these frozen directions exhibits positive curvature (sharpness). Updating these directions destabilizes the pre-trained solution, leading to limited adaptation gains or performance degradation.

3. Theoretical Analysis

The paper provides a theoretical justification for why PMP induces non-fine-tunability:

• Asymmetric Local Geometry: The pre-trained model reaches a conditional optimum where the loss landscape is flat along the trained subspace ($\theta_M$) but exhibits positive curvature along the frozen directions ($\theta_{\bar{M}}$).
• Instability: When fine-tuning without the mask, stochastic gradient descent updates both subspaces. Theoretical analysis (Proposition 1) demonstrates that updates to the frozen directions introduce destabilizing perturbations. Specifically, the expected pre-training loss increases (or fails to decrease) due to the positive curvature in the unadapted directions, counteracting any gains from task-specific gradients.
• Conclusion: Unauthorized fine-tuning is mathematically prone to destabilization, bounding the achievable fine-tuning gains.

4. Experimental Results

The authors evaluated PMP on GPT-2 and TinyLlama (1.1B) architectures, pre-trained on SlimPajama-6B and fine-tuned on the GLUE benchmark.

• Base Model Performance: PMP preserves the core capabilities of the base model. In many cases (e.g., TinyLlama on CoLA and MRPC), PMP models performed comparably to or slightly better than non-PMP baselines before fine-tuning.
• Unauthorized Fine-Tuning: When fine-tuned without the private mask, PMP models showed substantial degradation across diverse downstream tasks (e.g., SST-2, MRPC, QQP, MNLI). The performance drop was consistent across architectures and tasks, indicating a general failure of effective adaptation rather than task-specific brittleness.
• Authorized Fine-Tuning: When the private mask is provided to trusted users, they can fine-tune by updating only the masked subspace. In this scenario, PMP models matched or exceeded the performance of unauthorized fine-tuning, demonstrating that the method does not inherently prevent adaptation but rather restricts it to authorized parties.
• Ablation Studies:
  • Early-Bird vs. Random Mask: Using the Early-Bird lottery ticket mask was critical. Random masks of the same sparsity did not induce the same level of non-fine-tunability, confirming that the stability of the early-bird subnetwork is key.
  • Mask Ratio: A mask ratio of 0.7 (70% sparsity) offered the best trade-off between base performance and suppression of unauthorized fine-tuning.
  • Hyperparameters: Varying learning rates and fine-tuning epochs did not recover performance for unauthorized users, suggesting the limitation is fundamental to the optimization mismatch, not a hyperparameter tuning issue.
  • Loss Landscape Visualization: Visualizations confirmed that the loss landscape is flat in the authorized direction (masked subspace) but steep in the unauthorized direction (full parameter space), validating the theoretical "steep wall" hypothesis.
  • Robustness: The paper argues that gradient magnitudes alone do not reveal the mask location due to distributional overlap, making it difficult for adversaries to infer the mask.

5. Key Contributions

1. Problem Identification: The paper articulates non-fine-tunability as a fundamental pre-training-level problem, highlighting the lack of mechanisms for trainers to control unauthorized adaptation in open-weight releases.
2. PMP Framework: It proposes Private Mask Pre-Training, a general paradigm that preserves base utility while restricting unauthorized fine-tuning in a task-agnostic, black-box setting.
3. Theoretical Insight: It provides an analysis showing that PMP induces an intrinsic mismatch between pre-training and fine-tuning objectives, injecting destabilizing perturbations during unauthorized updates.
4. Empirical Validation: It demonstrates that PMP consistently suppresses adaptation gains across diverse benchmarks and architectures while maintaining competitive pre-trained performance.

6. Significance and Claims

The paper claims to reframe non-fine-tunability not as a task-specific unlearning problem, but as a property induced during pre-training. By concentrating learning into a private sparse subnetwork, PMP allows model trainers to open-source foundation models while retaining meaningful control over downstream adaptation.

The authors emphasize that this approach:

• Does not require prior knowledge of downstream data or tasks.
• Preserves the economic and safety value of the original model for the trainer.
• Allows for "authorized" adaptation by trusted users who possess the private mask.
• Offers a practical mechanism to reconcile the openness of the AI ecosystem with the need for safety and economic control, without relying on assumptions about how released models will be repurposed.