Basic syntax from speech: Spontaneous concatenation in unsupervised deep neural networks

1. Problem Statement

The paper addresses a fundamental gap in computational linguistics and cognitive science: how syntax (specifically concatenation and compositionality) emerges from raw acoustic inputs without explicit supervision or pre-defined symbolic rules.

• Current Limitations: Most computational models of syntax are text-based, rely on pre-trained symbolic representations, or use highly abstract theoretical assumptions. They fail to model the transition from holistic, single-unit signals (like animal calls) to concatenated, compositional signals (human language) directly from raw speech.
• The Core Question: Can domain-general deep learning models, trained solely on raw audio of single words, spontaneously learn to concatenate these words into multi-word sequences and exhibit traces of compositionality, mimicking the early stages of human language evolution and acquisition?

2. Methodology

The authors utilize ciwGAN (contextual information Word GAN) and fiwGAN (feature information Word GAN), which are information-theoretic extensions of Generative Adversarial Networks (GANs) designed for raw audio.

Model Architecture

• Generator (G): Takes latent codes ($c$, either one-hot or binary) and random noise ($z$) as input. It uses up-convolutional layers to generate audio (1.024s or 2.048s duration, 16kHz).
• Discriminator (D): Evaluates the "realness" of the generated audio against training data using Wasserstein loss.
• Q-Network: A crucial component that attempts to decode the latent code $c$ from the generated audio. This enforces informative imitation: the Generator must encode unique, decodable information into the audio to minimize the Q-network's error.
• Training Principle: The model learns via a production-perception loop. The Generator learns to produce speech that the Q-network can classify, mimicking human speech acquisition where learners imitate and refine based on feedback.

Experimental Design

Five experiments were conducted using the TIMIT database (raw spoken English words):

1. Models 1 & 2 (One-word, 1s): Trained on single words (5 and 10 types, respectively) with right-padding. Tested for spontaneous concatenation.
2. Models 3 & 4 (One-word, 2s): Trained on single words with random padding (silence) to 2 seconds. Model 4 used a complex setting with ~600 tokens per word.
3. Model 5 (Two-word, 2s): Trained on single words and some two-word combinations, but withheld a specific combination (e.g., "suit/greasy") to test generalization to unobserved pairs.

Key Constraint: The Generator never directly accesses the training data; it generates innovative outputs from noise. Concatenation must emerge spontaneously.

3. Key Results

A. Spontaneous Concatenation

• Negative Latent Codes: A robust pattern emerged where setting latent code values to negative numbers (outside the training range of 0 and 1) consistently triggered the generation of two or even three-word sequences.
  • Example: In Model 1, code [0, -2, -2, -2, 0] consistently produced "suit year."
  • Statistical Significance: Logistic regression showed a significant negative relationship between the sum of latent bits and the probability of single-word output ($p < 0.0001$). Negative values significantly increased the likelihood of multi-word outputs.
• Robustness: This phenomenon occurred across different architectures, hyperparameters, padding strategies (right-aligned vs. random), and batch sizes (even in batch size 1), suggesting it is an inherent property of the architecture rather than an artifact of training data alignment.

B. Generalization to Unobserved Combinations

• In Model 5, the model was trained on all combinations of three words except "suit/greasy."
• When tested, the Generator consistently produced the unobserved "greasy suit" (or "suit greasy") when specific latent codes were used. This demonstrates the model's ability to embed words into novel, unobserved combinations.

C. Semi-Compositional Concatenation

• The paper demonstrates that concatenation is not random but causally compositional for a subset of lexical items.
• Mechanism:
  • Positive extreme values (e.g., [1, 0, 0...]) represent a single word (e.g., "suit").
  • Negative extreme values (e.g., [0, -1.5, 0...]) represent a specific word (e.g., "greasy") often with a "weaker" acoustic profile or as a second element.
  • Composition: Combining a positive code for Word A and a negative code for Word B (e.g., [5, -1.5, 0...]) reliably produces the concatenated output "Word A Word B" (e.g., "suit greasy").
• This was observed in Model 2 with multiple word pairs (e.g., "year greasy," "ask greasy," "water greasy"), showing that the latent space organizes words such that their combination yields predictable concatenations.

4. Proposed Mechanism: Artificial Disinhibition

The authors propose a neural explanation for why negative latent values trigger concatenation, termed Artificial Disinhibition.

• Biological Analogy: In neuroscience, disinhibition occurs when an inhibitory neuron inhibits another inhibitory neuron, resulting in net excitation.
• Model Implementation:
  • The Generator's fully connected layer contains weights that can be positive (excitatory) or negative (inhibitory/silence).
  • During training, the model only sees non-negative latent codes ($0$ or $1$).
  • When a negative latent code is input during testing, it multiplies with the negative weights in the network.
  • Result: Negative $\times$ Negative = Positive. This "inhibition of inhibition" creates an excitatory signal that activates multiple word-shapes simultaneously, resulting in concatenated outputs.
• Formalization: The authors map this to a simplified version of the Merge operation (Chomsky).
  • $D(\gamma) = \{\alpha, \beta\}$: Disinhibition creates a set of two words.
  • $C(D(\gamma), \beta) = \{\alpha, \beta\}$: Combining a disinhibited representation with a standard representation yields a compositional concatenation.

5. Significance and Contributions

1. First Unsupervised Emergence of Syntax: This is the first study to demonstrate that concatenation and partial compositionality emerge spontaneously in a fully connectionist model trained on raw speech without explicit syntactic rules or multi-word training data.
2. Bridging Acoustics and Syntax: It challenges the view that syntax requires symbolic manipulation, suggesting instead that syntactic precursors can arise from the statistical learning of raw acoustic patterns via imitation.
3. Neural Plausibility: The "disinhibition" mechanism provides a biologically plausible pathway for how the brain might transition from holistic signals to compositional syntax, linking deep learning architectures to known neural processes (inhibitory/excitatory interactions).
4. Implications for Language Evolution: The findings support theories (e.g., Luuk & Luuk, 2014) that syntax evolved from a stage of simple signals to non-compositional concatenation, and finally to compositional syntax. The model replicates this trajectory.
5. Methodological Advancement: It validates the use of ciwGAN/fiwGAN as a tool for modeling language acquisition and evolution, offering a way to generate testable predictions for both artificial and biological neural processing of spoken language.

Conclusion

The paper successfully models the "first step" in the evolution of syntax: the shift from holistic single units to concatenated signals. By leveraging the unique properties of GANs (innovation via imitation) and the concept of disinhibition, the authors show that deep neural networks can spontaneously learn to combine words compositionally from raw audio, offering a new framework for understanding the origins of human language.