Connecting Speech to Words through Images

Technical Summary: Connecting Speech to Words through Images

Problem Statement
The paper addresses the challenge of learning the mapping between written words and their spoken counterparts without explicit textual supervision (transcripts). While visual grounding has been used to map speech to image regions or across languages, less emphasis has been placed on linking speech directly to written vocabulary. This capability is crucial for language documentation, particularly for low-resource languages, languages without a written form, or scenarios where collecting transcriptions is prohibitively expensive. The goal is to discover acoustic segments and assign them meaning in the form of written words using only a dataset of images paired with spoken descriptions.

Methodology
The proposed approach is a non-parametric, visually grounded method consisting of two primary stages: vocabulary building and keyword localization.

1. Vocabulary Building:

   • The system utilizes off-the-shelf image captioning models (specifically Tag2Text, BLIP-2, and GIT) to generate written descriptions for images in a visually grounded speech corpus.
   • Captions are lemmatized, and the most frequent words are selected to form a vocabulary. To ensure robustness, the final vocabulary is constructed from the intersection of words produced by all three captioning systems, filtering out visually ungrounded terms.
   • This creates a set of target words that have a verified visual correspondence within the dataset.

2. Keyword Localization:

   • Filtering: For a specific query word from the vocabulary, the system filters the speech corpus, retaining only those utterances whose associated image captions contain the query word.
   • Alignment: The selected utterances are aligned against one another to locate the recurring acoustic segment corresponding to the word. The paper explores two alignment variants based on HuBERT representations:
     • Discrete Features Alignment (DFA): HuBERT features are encoded into discrete units via k-means clustering. Pairs of unit sequences are aligned using the Smith-Waterman dynamic programming algorithm to produce a binary match signal.
     • Continuous Features Alignment (CFA): Alignment is estimated by computing cosine similarity directly in the continuous feature space. A smoothing filter is applied to reduce noise, and a threshold is used to binarize the signal.
   • Aggregation (Interval Piling): An utterance-wise score is computed by aggregating alignment scores between a specific utterance and all others in the filtered set. Segments with high aggregated scores (indicating frequent alignment across multiple utterances) are identified as containing the target word.
   • Segmentation: The speech is segmented based on contiguous regions bounded by silence or zero-valued frames, with final scores derived from the average frame-level values.

Key Contributions

• Visually Grounded Vocabulary Learning: The paper demonstrates a method to build a spoken word vocabulary and link it to written words using only images and speech, eliminating the need for manual transcripts.
• Novel Localization Framework: It adapts unsupervised word discovery techniques by constraining the search space to utterances filtered by visual context (image captions), rather than aligning the entire corpus.
• Comparative Analysis of Alignment Strategies: The study evaluates both continuous and discrete feature alignment methods, showing that while continuous methods offer higher performance ceilings, discrete methods are computationally more efficient and less dependent on captioning precision.
• State-of-the-Art Performance: The approach outperforms a strong neural baseline (an updated Attention CNN) in both keyword localization and keyword spotting tasks.

Experimental Results
Experiments were conducted on the MIT Places Audio Captions dataset (400k image-speech pairs), using a subset of 20k pairs. Ground truth was established using an ASR system (Parakeet TDT-CTC) and forced alignment (MMS model) for evaluation purposes only.

• Performance: The continuous feature alignment (CFA) method achieved the best results, with a Precision at Rank 10 (P@10) of 67.1% for keyword localization (IoU=0.5) and 35.4% for spotting. The discrete method (DFA) achieved 64.3% localization and 27.7% spotting.
• Baseline Comparison: Both visually grounded variants significantly outperformed the neural baseline (Attention CNN), which scored 44.4% (localization) and 25.1% (spotting).
• Topline Analysis: When using ground truth transcripts instead of image captions for filtering, performance improved significantly (up to 91.7% for CFA localization), highlighting the gap introduced by the imperfect alignment between image captions and actual speech.
• Ablation on Captioning: The study found that using the intersection of multiple captioning systems improved precision and downstream localization performance, suggesting that selecting fewer, highly accurate utterances is more effective than using a larger, noisier set.
• Failure Modes: Qualitative analysis revealed that failures often occur due to co-occurring words (e.g., "box" and "ring" in a "boxing ring" context) that are difficult to disambiguate without further context.

Significance and Claims
The paper claims to demonstrate the feasibility of connecting speech to written words without textual supervision, laying a foundation for future work on low-resource and unwritten languages. The authors position this work as a step toward bridging visual and spoken modalities for vocabulary discovery. They explicitly state that the technology aims to support language documentation efforts where transcriptions cannot be collected. The authors plan to extend this work to Hindi, Yoruba, and Japanese datasets, using English image captioners to filter utterances and uncover English-foreign speech associations. The ultimate goal is to develop technology that supports the preservation and documentation of local languages and cultural heritage.