Technical Summary: Neuro-Symbolic Lexical Discovery (NSLD)

1. Problem Statement

The paper addresses a critical bottleneck in deploying Large Language Model (LLM) agents in genuinely unknown environments (e.g., planetary or deep-sea exploration). While LLMs excel at natural language communication, they lack mechanisms to ground novel symbols on observable referents that have no name in any human language. Existing approaches either produce semantically opaque tokens or require direct human participation in vocabulary construction. This creates a "Symbol Grounding Problem" (SGP) where agents cannot acquire visual grounding on alien referents entirely absent from natural language corpora, nor can they associate these new "alien" symbols with known human words to report findings. The authors posit that for effective deployment in robotics, LLMs must be able to autonomously develop shared, grounded vocabularies for out-of-distribution (OOD) entities without prior linguistic representations.

2. Methodology: The Neuro-Symbolic Lexical Discovery (NSLD) Framework

The authors propose the NSLD framework, a neuro-symbolic system where a population of LLM-based agents plays a "Grounding Referential Game" to self-organize a shared alien lexicon.

2.1 Agent Architecture

Each agent consists of three integrated components:

• Perception (Vision): A frozen CLIP vision encoder (ViT-B/32) that projects visual inputs into a shared 512-dimensional embedding space. This module is decoupled from the reasoning engine to ensure transparent access to internal meaning vectors.
• Memory (Grounded Lexicon): A private FAISS vector index acting as a persistent lexical memory. It contains two partitions:
  1. A static perceptual base: ~3,000 English nouns/adjectives encoded via CLIP text, serving as a bridge to natural language.
  2. A dynamic alien word index: Private to each agent, storing CLIP image embeddings of visual referents associated with newly coined "alien" words.
• Reasoning (LLM): A text-only LLM (specifically gpt-oss:20b in experiments) responsible for decision-making, lexical selection, and game strategy. It interacts with the vision and memory modules via a controlled tool interface (e.g., observe_environment, add_alien_word, check_confidence).

2.2 The Grounding Referential Game

The framework operates through iterative rounds involving a Speaker and a Hearer:

1. Speaker Phase: The speaker observes a set of OOD visual referents (synthetic alien creatures, rocks, artifacts), selects one, and produces an "alien word" (a three-syllable string from a fixed phonological inventory, ensuring no overlap with natural language). The word is grounded by associating it with the CLIP embedding of the referent.
2. Hearer Phase: The hearer observes the same referent (joint attention) and must select a word from its own lexicon to refer to it.
3. Feedback & Update:
   • Success: If words match, both agents increase their confidence score for that word (+0.25).
   • Failure: If words differ, confidence decreases (-0.20). Agents then use their confidence scores to decide whether to retain their word or adopt the partner's word, updating their FAISS indices accordingly.

2.3 Experimental Setup

• Dataset: 10 synthetic OOD visual referents generated using Nano Banana Pro, designed to be unrecognizable to CLIP and the LLM training corpora.
• Scale: Simulations conducted with populations ($n_a$) ranging from 2 to 20 agents and entity counts ($n_e$) from 3 to 10.
• Constraints: Alien words are strictly constrained to a phonological inventory to prevent pre-trained linguistic shortcuts.

3. Key Results

3.1 Lexical Convergence

The system reliably achieves consensus across all tested configurations.

• Dynamics: Early rounds show high divergence with many unique, transient words. Over time, a "winner-take-all" dynamic emerges, leading to a unique shared alien word for each referent.
• Performance: In a 5-agent, 5-referent scenario, consensus (defined as $G \ge 0.95$) was reached after approximately 165 rounds.
• Scaling: Convergence time ($T_{0.95}$) scales monotonically with population size and entity count. The fastest configuration ($n_a=2, n_e=3$) converged in $16 \pm 9$ rounds, while the slowest ($n_a=20, n_e=10$) required $4572 \pm 472$ rounds.

3.2 Semantic Grounding and Human-Machine Alignment

Crucially, the discovered alien words are not semantically isolated. Because they are anchored to the CLIP embedding space, they inherit semantic proximity to the static English perceptual base.

• Interpretability: Each alien word is associated with coherent English descriptors (e.g., "rock," "creature") based on vector proximity, allowing human operators to understand the agent's vocabulary without observing the visual referents directly.
• Grounding: The system successfully creates "alien-to-English" links that are perceptually grounded, effectively enlarging the human vocabulary with new terms.

3.3 Analytical Modeling

The authors propose three mathematical models characterizing the convergence dynamics, all achieving $R^2 > 0.95$:

1. Direct Model: A logistic function predicting the degree of grounding consensus $G(T)$ at any round $T$ based on $n_a$ and $n_e$.
2. Inverse Model: Estimates the number of rounds required to reach a specific consensus threshold $\theta$.
3. Vocabulary Model: Predicts the evolution of global vocabulary size $V(T)$, capturing the initial explosion of unique words followed by a pruning phase as consensus is reached.

4. Key Contributions

1. Novel Framework for OOD Grounding: The NSLD framework demonstrates that LLM agents can acquire native perceptual grounding on referents with no prior linguistic representation, challenging the hypothesis that LLMs only possess indirect grounding via language corpora.
2. Autonomous Vocabulary Co-Development: The system achieves a shared human-machine vocabulary autonomously, without human intervention, solving a challenge previously identified as an open problem in human-machine interaction.
3. Interpretability via Embedding Space: By anchoring alien symbols to a shared embedding space, the framework renders emergent communication interpretable to humans through semantic proximity to natural language descriptors.
4. Predictive Planning Tools: The derived analytical models provide a first step toward pre-deployment planning for autonomous exploration missions, allowing resource estimation (rounds/time) based on population and environmental complexity.

5. Significance and Claims

The paper claims that NSLD represents a qualitative advancement over classical naming games and prior emergent communication studies. While previous works proved convergence to shared arbitrary tokens, NSLD proves convergence to shared grounded symbols that are decoupled from conversational context and anchored in persistent perceptual memory.

The authors position this work as a concrete engineering solution to the Symbol Grounding Problem in robotics, moving beyond philosophical debate to demonstrate that LLMs can acquire novel grounding natively. The significance lies in enabling autonomous agents to operate in truly unknown environments, develop a shared lexicon for unnameable entities, and report these findings to humans using a semantically grounded vocabulary. The authors acknowledge limitations, noting that experiments are currently simulation-based, limited to a single LLM backend, and utilize fully connected population topologies, identifying these as immediate directions for future work.