HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation

This paper proposes HIERAMP, a method that leverages the coarse-to-fine generation capability of Vision Autoregressive (VAR) models to amplify hierarchical semantics through dynamic class token injection, thereby improving dataset distillation performance by better capturing object structures and details without explicitly optimizing global proximity.

The Gist

Imagine you are trying to teach a student how to recognize different animals, but you only have a tiny budget for textbooks. You can't buy the entire library (the massive original dataset), so you need to create a "super-condensed" cheat sheet (a distilled dataset) that fits on a single page but still teaches the student everything they need to know.

This is the problem of Dataset Distillation.

For a long time, researchers tried to make these cheat sheets by just making them look "statistically similar" to the real library. It's like trying to teach someone what a "dog" is by showing them a blurry, averaged-out picture of a dog. It captures the general vibe, but it misses the specific details that make a dog a dog (like the shape of its ears or the texture of its fur).

Enter HIERAMP: The "Zoom-In" Teacher

The authors of this paper, HIERAMP, realized that learning happens in layers. You don't learn to draw a bird by starting with the tiny feathers; you start with the big shape of the body, then the wings, and finally the details.

They built a system that mimics this natural learning process using a technique called Visual Autoregressive (VAR) generation. Think of VAR as an artist who paints a picture in stages:

1. Coarse Stage: They sketch the rough outline and placement of objects (e.g., "There's a bird in the sky").
2. Fine Stage: They add the feathers, the beak, and the eye details.

The Problem:
Standard methods often get stuck trying to match the "average" look of the data. They might produce a dataset that looks like a dog, but it lacks the specific features that help a computer tell the difference between a Golden Retriever and a Labrador.

The HIERAMP Solution: "Amplifying the Important Bits"

HIERAMP acts like a smart teacher who knows exactly where to look. Here is how it works, using a simple analogy:

1. The "Class Token" (The Spotlight)

Imagine the AI is looking at a picture of a bird. HIERAMP attaches a special "Spotlight Token" to the image. This token is like a teacher's finger pointing at the most important parts of the picture.

• In the Coarse Stage, the teacher points at the general shape: "Look at the body and the wings!"
• In the Fine Stage, the teacher points at the details: "Look at the eye and the beak!"

2. The "Amplification" (Turning Up the Volume)

Once the teacher points to the important spots, HIERAMP amplifies them.

• At the beginning (Coarse): It makes the "big picture" choices more diverse. Instead of just drawing one type of bird body, it encourages the AI to try many different body shapes and positions. This ensures the student learns the structure of the object, not just one specific pose.
• At the end (Fine): It focuses the attention intensely on the details. It tells the AI, "Don't waste time on the background; make the feathers and eyes super sharp and distinct."

3. The Result: A Better Cheat Sheet

By doing this "Coarse-to-Fine" amplification, HIERAMP creates a tiny dataset that is incredibly rich in information.

• Without HIERAMP: The cheat sheet might have 10 pictures of birds that all look exactly the same (boring and unhelpful).
• With HIERAMP: The cheat sheet has 10 pictures of birds that show different angles, different body shapes, and very clear, sharp details.

Why is this a big deal?

Usually, making a dataset smaller means losing quality. HIERAMP proves you can shrink the dataset without losing the "soul" of the data.

• It's efficient: It doesn't require heavy, slow processing. It's like adding a filter to a camera rather than rebuilding the whole camera.
• It's smart: It understands that "structure" (the skeleton of the object) and "details" (the skin and texture) need different kinds of attention.

In Summary:
HIERAMP is like a master chef who knows that to make a perfect soup (the distilled dataset), you first need to get the broth right (coarse structure) and then season it perfectly (fine details). Instead of just mixing everything together randomly, they taste the soup at every stage and add extra spice to the most important flavors. The result? A tiny spoonful of soup that tastes just as good as a whole pot.

Technical Summary

Here is a detailed technical summary of the paper "HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillation."

1. Problem Statement

Dataset Distillation (DD) aims to synthesize a small surrogate dataset from a large training corpus that preserves the performance of models trained on the original data.

• Current Limitations: Most existing methods focus on global semantic proximity (matching feature distributions or training trajectories between synthetic and real data). They often treat images as monolithic entities in pixel or latent space.
• The Gap: Object semantics are inherently hierarchical. For example, the position of a bird's eyes is constrained by the head outline, which is constrained by the body. Global proximity fails to capture how structures at different levels (coarse layout vs. fine details) support recognition. Consequently, distilled datasets often lack the discriminative, class-specific details necessary for high downstream performance, appearing as feature abstractions rather than natural images.

2. Methodology: HIERAMP

The authors propose HIERAMP, a framework that leverages Visual Autoregressive (VAR) models to perform dataset distillation via coarse-to-fine semantic amplification.

Core Architecture: Visual Autoregressive (VAR)

Unlike standard autoregressive models that predict the next token, VAR predicts the next scale of the image.

• It generates images hierarchically: Scale 1 (coarse layout) $\to$ Scale 2 (mid-level structure) $\to$ ... $\to$ Scale N (fine details).
• This structure naturally aligns with the hierarchical nature of object semantics.

Key Components of HIERAMP

1. Scale-Restricted Class Token Attention

• Injection: A learnable class token is injected into the attention mechanism at each scale of the VAR model.
• Constraint: Unlike standard tokens that attend to previous scales, the class token at scale $n$ is masked to attend only to tokens within the same scale $n$.
• Function: This forces the class token to aggregate a "scale-specific semantic summary." It learns to identify which spatial regions at that specific resolution are most relevant to the class label.
• Training: The class token is optimized using a classification objective to ensure it captures discriminative semantics.

2. Semantic Saliency Mapping

• The attention weights of the class token at scale $n$ are aggregated to form a saliency map ($M_n$).
• This map highlights regions with high "object-related semantics" for that specific scale (e.g., global shape at coarse scales, texture at fine scales).

3. Coarse-to-Fine Autoregressive Amplification

• Mechanism: During the autoregressive decoding process, HIERAMP identifies the top $\rho\%$ most salient positions in the saliency map.
• Amplification: A positive logit bias ($\beta$) is added to the attention logits for these salient keys. This steers the model's attention toward discriminative regions during generation.
• Stage-Aware Strategy: The amplification is applied differently across stages:
  • Coarse Scales (1–3): Amplification encourages diversity in token choices, ensuring rich global layouts and object placement.
  • Fine Scales (7–9): Amplification concentrates token usage, focusing on refining specific object details and textures.
  • Mid Scales: Balanced amplification to bridge structure and detail.

3. Key Contributions

1. Hierarchical Perspective: Shifts the dataset distillation paradigm from global distribution matching to hierarchical semantic modeling, recognizing that object semantics exist at multiple scales.
2. Novel Framework (HIERAMP): Introduces a method to inject learnable class tokens into VAR models to dynamically identify and amplify salient semantic regions at every generation scale.
3. Efficiency: The method adds only marginal inference cost (no external segmentation tools or heavy guidance at test time) while significantly improving synthesis quality.
4. Insight into Token Dynamics: Provides empirical evidence that:
   • Amplifying coarse scales increases token entropy and diversity (better global structure).
   • Amplifying fine scales decreases entropy (more focused, repetitive details).
   • The most significant accuracy gains come from amplifying coarse scales, as they set the structural foundation for subsequent details.

4. Experimental Results

The method was evaluated on standard dataset distillation benchmarks (CIFAR-10/100, ImageNet-Woof, ImageNet-100, ImageNet-1K) with varying Images-Per-Class (IPC) settings (1, 10, 50, 100).

• State-of-the-Art Performance: HIERAMP consistently outperforms existing SOTA methods (Minimax, D3HR, RDED, CaO2) across almost all datasets and architectures (ResNet-18, ResNet-101, MobileNet-V2, EfficientNet-B0).
  • Example: On ImageNet-1K (IPC=10) with ResNet-18, HIERAMP achieved 47.6% accuracy, surpassing the second-best method (CaO2) by 1.5%.
  • Example: On ImageNet-1K (IPC=50), it reached 66.4% accuracy.
• Cross-Architecture Generalization: Distilled datasets generated by HIERAMP show strong transferability, achieving high accuracy when training diverse student networks, even when the teacher network differs.
• Generative Quality:
  • FID Scores: HIERAMP achieves lower Fréchet Inception Distance (better image quality) compared to diffusion-based distillation methods.
  • Latency: It is significantly faster than diffusion-based methods (0.147s/img vs. 0.456s/img for 30-step DDIM) due to the efficient VAR architecture.
• Ablation Studies: Confirmed that amplifying coarse scales yields the highest performance gains. Balancing amplification across stages (Coarse-Mid-Fine) is crucial for optimal results.

5. Significance

• Explainability: HIERAMP offers a new lens to understand dataset distillation by analyzing token distributions (entropy and coverage) across hierarchical scales. It reveals that "richness" in global structure is more critical for downstream performance than previously thought.
• Trustworthy AI: By explicitly modeling and amplifying discriminative object semantics rather than just matching global statistics, the resulting distilled datasets are more representative and robust, leading to more trustworthy downstream models.
• Scalability: The approach is computationally efficient and scales well to large datasets like ImageNet-1K, addressing a major bottleneck in current generative distillation methods.

In summary, HIERAMP demonstrates that effective dataset distillation requires not just matching data distributions, but actively amplifying hierarchical semantic structures during the generative process, leading to superior synthetic datasets that better support model training.