🤖 machine learning

Communicating Sound Through Natural Language

This paper introduces Lexical Acoustic Coding (LAC), a framework where pre-trained LLM agents communicate sound by converting waveforms into interpretable English text descriptions and reconstructing them through closed-loop refinement, effectively treating natural language as a finite-rate, lossy transport representation for audio.

Original authors: Emanuele Rossi, Emanuele Rodolà

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emanuele Rossi, Emanuele Rodolà

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to send a recording of a sound—like a drum hit or a bell chime—to a friend. Usually, you'd send the actual audio file (a waveform), which is like sending a heavy, bulky box full of raw data.

This paper introduces a new way to send sound called Lexical Acoustic Coding (LAC). Instead of sending the heavy box, you send a letter.

Here is how the system works, broken down into simple steps:

1. The "Translator" (The Sender)

Imagine you have a sound, like a "punchy, warm drum hit."

The Analysis: A computer program (the sender) listens to the sound and breaks it down into 47 specific, measurable traits. It doesn't guess; it measures things like "how loud is the peak?" (RMS energy), "how fast does it start?" (attack time), and "what is the pitch?" (fundamental frequency).
The Dictionary: The system has a shared dictionary of words for these measurements. For example, instead of sending the number "0.25," it looks up the word "mid-power." Instead of "0.04 seconds," it uses "staccato."
The Letter: The computer takes all 47 of these words and writes them into a normal English sentence.
- Example: "The sound hits with a mid-power punch and a staccato decay. Its spectrum is warm and spread..."
- This sentence is the only thing sent over the internet. No audio files, no secret codes, just plain text.

2. The "Receiver" (The Decoder)

Your friend receives the sentence.

The Reverse Translation: A second computer program reads the sentence and pulls out the 47 words. It knows that "mid-power" means the volume should be between 0.10 and 0.30.
The Guess: It takes these ranges and tries to build a sound that fits the description. It's like a chef trying to recreate a dish based only on a written recipe, without ever tasting the original.
The "Taste Test" (Refinement): The computer makes a first guess at the sound. Then, it listens to its own creation and checks: "Does this sound like 'staccato'? Is it 'warm'?" If the answer is "not quite," it tweaks the knobs and tries again. It does this loop a few times until the sound matches the description in the letter as closely as possible.

What is the Result?

The final sound isn't an exact, pixel-perfect copy of the original (like a photocopy). Instead, it is a reconstruction.

If you send a snare drum, you get a snare drum back.
If you send a "warm, metallic clang," you get a sound that feels warm and metallic.
The paper shows that while the waveforms don't match exactly, the vibe, the rhythm, and the texture are preserved.

Why is this special?

The authors compare this to other ways of handling sound:

Standard Audio Files (WAV/FLAC): These are like sending the original painting. They are perfect, but you can't read them, and they are huge.
Neural Codecs (AI compression): These are like sending a compressed digital file. They are small, but the data is a secret code only machines understand. You can't edit them easily.
Captions: These are like a description ("A loud dog barks"). They are easy to read, but they are too vague to rebuild the sound.

LAC sits in the middle. It is a readable code.

Human-Readable: You can read the sentence and understand what the sound should be like.
Editable: If you want the sound to be "louder" instead of "mid-power," you can just change that one word in the sentence, and the receiver will build a louder sound.
Machine-Readable: Computers can read the sentence and rebuild the sound without needing a special, trained AI model for the specific file.

What are the limits?

The paper is very clear about what this system cannot do yet:

It is not for long songs or speech. It works best on short, isolated sounds (like a drum hit, a pluck, or a short note).
It is not a perfect copy machine. It recreates the character of the sound, not the exact mathematical waveform.
It currently handles the "timbre" (the color of the sound) well, but if you want to send a whole song, you still need a separate system to send the musical notes (the melody and rhythm), while LAC just sends the "instrument sound."

In short, the paper proves that we can turn sound into a descriptive sentence, send that sentence, and have a computer rebuild a very similar sound on the other end, all without sending a single byte of actual audio data.

Technical Summary: Communicating Sound Through Natural Language (Lexical Acoustic Coding)

1. Problem Statement

Natural language is currently peripheral to audio systems, serving primarily as metadata, prompts, or captions, while the audio itself is transported via waveforms, continuous latents, or learned codec tokens. This paper investigates a distinct design point: can natural language serve as the transport representation for sound itself?

The core challenge is to determine whether a sufficiently structured vocabulary can carry enough measurable acoustic information to support a usable round-trip communication between language-model agents. Unlike conventional codecs that prioritize bit-level invertibility, or captions that prioritize human readability but lack reconstruction fidelity, this work seeks a representation that is simultaneously human-readable, acoustically interpretable, and capable of enabling lossy waveform reconstruction.

2. Methodology: Lexical Acoustic Coding (LAC)

LAC is a framework where pre-trained Large Language Model (LLM) sender and receiver agents transmit sound through natural language. The system operates under fixed system prompts where agents write their own analysis and synthesis code, communicating solely through a lexical sentence, a shared vocabulary, and optional symbolic music structure.

2.1 The Pipeline

The process follows a communication protocol with a shared vocabulary:

Shared Vocabulary Setup: A vocabulary $V$ is established, containing a feature set $\mathcal{F}$ (e.g., spectral centroid, decay time) and corresponding lexical alphabets $\mathcal{A}_i$ for each feature. This vocabulary can be human-authored or generated by an agent.
Encoding (Sender):
- Feature Extraction: An input waveform is analyzed into a $d$ -dimensional vector of acoustic features using off-the-shelf tools (e.g., librosa). The system uses 47 features covering temporal, spectral, harmonic, and psychoacoustic domains.
- Lexical Quantization: Each feature value is mapped to a discrete lexical label via a feature-specific interval table (e.g., an RMS value in $[0.10, 0.30)$ maps to "mid-power"). This creates a lexical code $\ell = (\ell_1, \dots, \ell_d)$ .
- Verbalization: The code is converted into a structured English sentence. The verbalizer ensures the sentence is readable but remains an injective carrier; it cannot delete, merge, or ambiguously rename terms.
Transmission: Only the English sentence is transmitted. No binary audio payload, learned latent, or non-ASCII side channel is used.
Decoding (Receiver):
- Parsing: The receiver parses the sentence back into the $d$ -label lexical code.
- Label Inversion: Each label is inverted to a representative numeric value (typically the interval midpoint) and an interval constraint.
- Synthesis: A deterministic hybrid synthesizer (combining harmonic, modal, and noise-based components) generates a waveform based on these targets.
- Closed-Loop Refinement: The system re-analyzes the synthesized waveform, scores it against the transmitted interval constraints, and iteratively adjusts renderer controls (e.g., envelope scaling, modal density) to minimize mismatch.

2.2 Theoretical Framing

LAC is framed as a finite-rate lossy quantizer. The information capacity is bounded by the vocabulary size ( $B_{max} = \log_2 |\mathcal{L}|$ ). The system trades bit-level invertibility for properties like human readability and text-native transport, operating closer to a semantic communication system than a conventional codec.

3. Key Contributions

The paper makes three primary contributions:

Formalization of LAC: A framework that quantizes acoustic descriptors into a short lexical code, verbalizes it for plain-text transmission between LLM agents, and provides finite-rate and lossy-quantizer results for the channel.
Agentic Setting: An experimental setup where sender and receiver agents write their own analysis and synthesis code under fixed prompts, communicating only through the lexical sentence and shared vocabulary, without being trained specifically for the task.
Hybrid Reconstruction: The introduction of interval-aware closed-loop refinement, demonstrating that LAC supports both short isolated sounds and symbolic music transfer (where structure is preserved externally via ABC notation and timbre is carried by LAC).

4. Experimental Results

Experiments were conducted using GPT-5.3-Codex agents on a dataset of 3,707 short audio samples (0.0002s to 2s) derived from tracker music modules.

Feature Family Ablation: Pre-synthesis accuracy (inverting labels to values) reached 100% with all 47 features. Post-synthesis accuracy (reconstructing the waveform and re-extracting features) reached approximately 74% with all features included. Bark-band information provided the largest gain, while temporal and psychoacoustic families added marginal improvement.
Closed-Loop Refinement: Refinement significantly improved reconstruction. Accuracy rose from ~72% (no refinement) to ~84% with 64 evaluations. Throughput decreased inversely with the number of evaluations.
Qualitative Performance: Reconstructions did not match the original waveforms sample-for-sample. However, they preserved macroscopic envelopes, dominant periodicity, and the tonal-versus-noisy balance, resulting in perceptual similarity.
Music Transfer: The system successfully transferred timbre while preserving symbolic structure (notes, timing) transmitted separately in ABC notation.

5. Significance and Scope

The paper positions LAC as a distinct point in the design space of audio representations:

Not a Codec Replacement: LAC is not designed to replace neural codecs (like EnCodec or SoundStream) on raw rate–distortion grounds. It does not support exact sample recovery, speech coding, or full music compression.
Interpretable Transport: It offers a human-readable, inspectable, plain-text acoustic representation. Humans can read and audit the representation; language models can manipulate it; and decoders can render it into sound.
Semantic Communication: It bridges the gap between captions (readable but weak for reconstruction) and codecs (invertible but opaque). The transmitted sentence serves as both a rich caption and the transport representation itself.

Limitations:

The system targets short, isolated, non-speech sounds (hits, bursts, plucks). It struggles with sustained, time-varying sounds and does not capture phonetic content, speaker identity, or long-range musical structure (melody, harmony) without external symbolic representation.
Decoding is inherently approximate; it reconstructs sounds consistent with a lexical description rather than recovering the original waveform exactly.
Symbolic music transfer currently relies on ABC notation; describing complex musical structures in unconstrained prose remains a constraint-satisfaction problem.

In conclusion, LAC demonstrates that plain text can preserve measurable acoustic structure while remaining interpretable and editable, offering a new paradigm for agent-mediated sound communication.