Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgetting

1. Problem Statement

The paper addresses a fundamental limitation in modern deep learning: data scarcity. While pre-trained weights are crucial for transfer learning, obtaining "better" weights typically requires larger pre-training datasets, which are often expensive or impossible to collect due to computational and curation costs.

The authors pose two key questions:

1. What defines a "better" pre-trained weight? (Answer: Weights trained on larger datasets generally yield better generalization).
2. How can we obtain these better weights without actually collecting more data?

Existing methods like Weight Averaging or Task Vectors attempt to combine knowledge but do not explicitly model the process of forgetting to recover lost information. The paper hypothesizes that forgetting (loss of knowledge when fine-tuning on a subset of data) is not merely a drawback but a structured process that can be inverted to synthesize weights containing more knowledge than the original training set.

2. Methodology

The proposed framework, KNOW (KNowledge-Overflowed Weights) Prediction, operates on the principle of Retrodiction of Forgetting.

A. Core Concept: Progressive Forgetting and Inversion

The method constructs a controlled forgetting trajectory:

1. Sequential Downsampling: Start with a dataset $D_0$ and a pre-trained model $\Theta_0$.
2. Forgetting Steps: Iteratively fine-tune the model on progressively smaller subsets ($D_1 \subset D_0, D_2 \subset D_1, \dots$). This induces a sequence of weights $[\Theta_0, \Theta_1, \Theta_2, \dots, \Theta_S]$ where knowledge about the excluded data is progressively "forgotten."
3. Retrodiction: The goal is to reverse this trajectory. By analyzing the transition from $\Theta_S$ back to $\Theta_0$, the model predicts a hypothetical weight state $\Theta_{-1}$ that corresponds to training on a virtual, larger dataset ($D_{-1}$ where $D_0 \subset D_{-1}$).

B. The KNOWN Model (Meta-Hypernetwork)

To perform this prediction, the authors introduce KNOWN (KNowledge Overflowed Weights Nowcaster), a lightweight meta-learned hyper-model.

• Architecture: A two-stream MLP (Multi-Layer Perceptron) inspired by the Weight Nowcasting Network (WNN), containing only ~9,425 parameters.
• Input: It takes the current weight state ($W^t$) and the weight updates ($dW^t$) from a sequence of $S$ forgetting steps.
• Output: It predicts the residual update required to reach the "knowledge-enriched" state ($\hat{\Theta}_{-1}$).
• Training: KNOWN is meta-trained on a dataset of weight transitions generated from various small-scale architectures (CNN, ResNet, etc.) and datasets (CIFAR, MNIST) under different sampling rates. The objective is to minimize the $L_1$ residual error between the predicted weight and the true weight of the previous (larger) dataset step.

C. Iterative Multi-step Forecasting

The method supports iterative prediction. Once $\hat{\Theta}_{-1}$ is predicted, it can be used as a base to predict $\hat{\Theta}_{-2}$, effectively extrapolating the knowledge capacity multiple times (e.g., $\times 2, \times 4, \times 8$ scaling of the effective dataset size).

3. Key Contributions

1. KNOW Prediction Strategy: A novel paradigm that treats forgetting as a reversible, structured process to synthesize "knowledge-overflowed" weights without additional data collection.
2. Weight Transition Dataset: Construction of a large-scale dataset of weight trajectories governed by controlled sequential forgetting to facilitate meta-learning.
3. KNOWN Hyper-model: A lightweight, meta-trained predictor capable of generalizing across diverse architectures and tasks to predict weights trained on virtual, larger datasets.
4. Empirical Validation: Comprehensive experiments demonstrating that these predicted weights outperform standard fine-tuning and other weight-prediction baselines.

4. Experimental Results

The authors evaluated KNOW prediction across diverse tasks and architectures (ResNet18, PVTv2, DeepLabV3+) using datasets like CIFAR-10/100, ImageNet, PACS, and Cityscapes.

• Image Classification (CIFAR-10/100):

  • Using ResNet18, KNOW prediction consistently outperformed Naïve Transfer, Incremental Pre-training, and curve-fitting baselines (Linear, Log, Exp).
  • Key Finding: With a $\times 8$ prediction factor (simulating training on 8x the data), the method achieved 93.55% accuracy on CIFAR-10, significantly surpassing the baseline (92.40%) and even outperforming models trained on the full dataset.
  • Baselines like "Task Vector" or "LogFit" often failed or degraded performance when the extrapolation distance increased, whereas KNOWN remained robust due to its learned non-linear trajectory modeling.

• Domain Generalization (PACS Dataset):

  • In a Leave-One-Domain-Out setting, KNOWN improved average accuracy by +4.73% over the baseline, demonstrating superior adaptability to unseen domain styles (Art, Cartoon, Photo, Sketch).

• Novel Modalities (Image Captioning & Segmentation):

  • Flickr8K (Captioning): Improved masked accuracy by ~2.2% on a novel text-generation task.
  • Cityscapes (Semantic Segmentation): Improved mIoU by +2.70% at the $\times 9$ scaling factor, whereas Task Vector failed in this regime. Qualitative results showed better detection of small objects (e.g., traffic lights) and reduced segmentation instability.

• Efficiency:

  • The prediction step is computationally negligible (<3 seconds for ResNet18) as it requires only inference.
  • The total training cost is reduced compared to collecting new data, as the method simulates the benefits of large-scale training using a fraction of the data via the forgetting process.

5. Significance and Conclusion

This paper fundamentally reinterprets forgetting in deep learning. Instead of viewing it as a loss to be avoided, the authors leverage it as a signal to reconstruct and enhance knowledge.

• Theoretical Insight: It validates that weight trajectories induced by progressive forgetting are smooth and reversible, allowing for the extrapolation of "virtual" weights that encapsulate more knowledge than the physical training data permits.
• Practical Impact: KNOW prediction offers a cost-effective solution for data-scarce scenarios. It allows practitioners to generate high-quality, knowledge-enriched initializations for downstream tasks without the prohibitive cost of large-scale data collection or pre-training.
• Generalizability: The KNOWN model demonstrates strong zero-shot transfer capabilities, working effectively across different architectures (CNNs, Transformers) and tasks (classification, segmentation, captioning) without requiring re-meta-training.

In summary, the work proposes a "time-travel" mechanism for neural networks: by intentionally forgetting data in a controlled manner, the model learns to predict the state it would have been in had it seen more data, thereby pushing the limits of knowledge transfer.