Technical Summary: Wake Vision

Problem Statement

Tiny Machine Learning (TinyML) involves deploying models on microcontrollers (MCUs) with extreme memory constraints (a few hundred kilobytes), necessitating small models and tailored binary tasks or few-class tasks. However, general practices for datasets in this domain are unsuitable. Small models are disproportionately sensitive to label noise, and tailored tasks often lack standardized benchmarks.

The previous standard for person detection in the TinyML domain, the Visual Wake Words (VWW) dataset, contains approximately 123,000 images with an estimated label error rate of 7.8%. This scale and quality are insufficient for training TinyML systems for production deployment. Conversely, manual labeling of datasets at the scale required for modern training (millions of images) is prohibitively expensive and leads to inconsistencies. For example, the cost of manually relabeling a 6-million-image dataset is estimated at approximately $600,000.

Methodology: The Wake Vision Pipeline

To address these gaps, the authors propose the Wake Vision Pipeline, an automated method for generating and curating large-scale binary classification datasets specifically for TinyML. The pipeline is data-centric by design and targets the entire lifecycle of a dataset, from creation to continuous improvement. It is based on the Open Images v7 dataset, which offers 9,600 trainable classes, enabling the pipeline to be adapted for any binary classification task, not just person detection.

The pipeline executes six data-centric core operations:

1. Multi-Source Label Fusion: It merges image-level and bounding-box annotations from Open Images into a single binary task. It leverages the higher accuracy of bounding-box labels while simultaneously exploiting the scalability of image-level labels.
2. Context-Aware Filtering: It applies filtering based on confidence scores, the area of the object in the image (a proxy for distance), and representation status (artistic representations of people are excluded by default). This ensures the dataset aligns with deployment conditions where distant or depicted objects may not be relevant.
3. Automated Label Correction: It employs Confident Learning techniques on evaluation splits to identify potential errors as a cost-effective first pass.
4. Manual Relabeling of Evaluation Sets: To ensure accurate performance assessment, the validation and test sets are manually relabeled via a crowdsourcing platform (Scale AI). This prioritizes label quality where it most strongly influences reported metrics.
5. Fine-Grained Benchmark Synthesis: Using metadata from the filtering steps, the pipeline generates evaluation subsets across five dimensions: perceived gender, perceived age, distance, lighting, and representations.
6. Community-Driven Improvement: The pipeline supports a "flywheel" model where community competitions identify and integrate improvements (e.g., label correction techniques) into subsequent dataset releases.

Key Contributions

1. The Wake Vision Dataset

The authors release Wake Vision, a binary dataset for person detection containing nearly 6 million images (approximately 100 times more person images compared to VWW).

• Scale: 5.76 million images in the "Large" training set and 1.25 million in the "Quality" training set.
• Quality: The validation and test sets were manually relabeled, achieving a label error rate of 2.2%, a significant improvement over the 7.8% of VWW.
• Licensing: Released under CC-BY 4.0 via TensorFlow Datasets and Hugging Face.

2. A Fine-Grained Benchmark Suite

Beyond aggregate accuracy, Wake Vision offers five fine-grained benchmark subsets to evaluate model robustness in challenging real-world scenarios:

• Perceived Gender & Age: Evaluates performance across underrepresented demographic groups.
• Distance: Categorizes subjects by image coverage (Near >60%, Medium 10–60%, Far <10%).
• Lighting: Categorizes by average pixel value (Dark, Normal, Bright).
• Representations: Tests the model's ability to reject images containing depictions of people (e.g., paintings) versus real people.

3. Generalization to Other Tasks

The pipeline is demonstrated for Car- and Bird Detection.

• Car Detection: Generated a dataset 12 times larger than a VWW-like baseline and reduced the label error rate from 6.4% to 4.8%.
• Bird Detection: Generated a dataset 27 times larger than a VWW-like baseline and reduced the label error rate from 6.6% to 0.6%.

4. Community Improvement Loop

The authors collaborated with the Edge AI Foundation to host the Wake Vision Challenge. The first round yielded a label correction technique that reduced the label error rate of the Wake Vision (Large) training set from 15.2% to 9.8% at a fraction of the cost of manual relabeling.

Results

Performance Improvements

Models trained on Wake Vision consistently outperform models trained on VWW across various architectures (MobileNetV2, MCUNet, MicroNets, ColabNAS):

• Accuracy: Models trained on Wake Vision improved test accuracy by up to 6.6% compared to models trained on VWW.
• Fine-Grained Robustness: Models trained on Wake Vision achieved equal or higher performance on 13 out of 16 fine-grained subsets compared to models trained on VWW. The largest gains were observed in detecting depicted people, older people, and under dark lighting conditions.
• Out-of-Distribution (OOD) Generalization: Models trained on Wake Vision maintained their accuracy advantage over models trained on VWW when evaluated on OOD datasets covering driving perspectives and aerial surveillance footage.

Data-Centric Insights

The work presents two specific insights regarding data-centric TinyML:

1. Sensitivity to Label Noise: Small models are disproportionately sensitive to label noise in the training set. When the training error rate increased from 6.8% to 15%, the smallest model lost 1.3% accuracy, whereas the largest model lost only 0.5%.
2. Two-Stage Training Strategy: A strategy of pre-training on the larger, noisier set (Wake Vision Large) and fine-tuning on the smaller, cleaner set (Wake Vision Quality) achieved the best results (85.72% accuracy), outperforming training on either set alone.

Significance and Claims

The work claims that Wake Vision closes the methodological gap in the TinyML domain by providing an automated pipeline for generating datasets for production deployment. Its significance lies in:

• Scalability: Enabling the creation of tailored datasets for the 9,600 classes in Open Images without prohibitively high costs for manual labeling.
• Quality: Demonstrating that manual relabeling of evaluation sets combined with automated cleaning yields datasets suitable for training production systems, not just research benchmarks.
• Evaluation: Shifting the focus from aggregate accuracy to fine-grained robustness, thereby uncovering failure modes (e.g., in low light or for specific demographics) that standard metrics obscure.
• Sustainability: Establishing a model for continuous dataset improvement through community engagement, treating the dataset as a maintained artifact rather than a static publication.

The authors position Wake Vision not merely as a dataset, but as a toolkit for the "data-centric TinyML" paradigm, arguing that for small models, returns from label quality dominate earlier than scaling does, as is the case with training large models, necessitating rigorous curation pipelines.