💬 NLP

When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models

This paper introduces UniLang, a unified generative framework that extends pretrained large language models to natively process and generate machine-native symbols alongside natural language, thereby bridging the gap between linguistic modeling and structured prediction without requiring verbalization or task-specific architectures.

Original authors: Su Yan, Rakesh Iyer

Published 2026-08-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Su Yan, Rakesh Iyer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of artificial intelligence, a quiet tension has long existed between two ways of describing the world. On one side sits the language we speak, a fluid and rich medium that humans use to share stories, ideas, and feelings. Large language models, the powerful computer programs that can write essays or answer questions, are trained on this human language. They understand the world through words. On the other side sits the language of machines, a rigid and compact system of codes and numbers. Computers often store complex information, like a specific movie or a legal ruling, not as a paragraph of text, but as a short, efficient string of symbols. These symbols are perfect for calculation and storage, but they are invisible to the language models that rely on human words. For years, researchers have had to choose between these two worlds. To make a computer understand a machine code, they often had to translate it into a sentence first, which could lose important details. Or they had to build special, separate systems just for the codes, leaving the powerful knowledge of the language models behind.

A team of researchers at Google has now proposed a way to bring these two worlds together without forcing a translation. They introduced a new framework called UniLang, which allows a standard language model to treat machine codes as if they were words. Instead of converting a machine symbol into a sentence like "a movie from 1999," the model learns to generate the symbol itself as a direct part of its output, just like it would generate the word "movie." The researchers tested this idea on two very different problems: predicting what a person might watch next based on their history, and finding the exact sentence in a past court case that supports a new legal argument. In both cases, the new method worked better than existing systems that either relied on pure text or pure codes. By letting the model speak in both human language and machine symbols at the same time, the researchers showed that artificial intelligence can handle structured data more accurately while still using the vast knowledge it learned from reading human books and websites.

The core of this work lies in how the researchers taught the computer to see these machine symbols. Imagine a library where every book has a unique barcode. For a long time, a librarian who only speaks English could not understand the barcode; they would have to read the title on the cover to know what the book was. The new method gives the librarian a new set of tools. They take the barcode and teach the librarian that this specific pattern of lines means "the book about space travel," not by reading a description, but by learning the direct connection between the pattern and the object. In the study, the researchers took thousands of items, such as movies or legal cases, and turned them into these compact machine codes. They then used a learning process to align these codes with the text descriptions of the same items. This meant that when the computer saw the code for a specific movie, it understood it in the same way it understood the words "action movie from the 1980s."

Once the computer learned this new vocabulary, the researchers asked it to perform tasks that usually require two different types of thinking. In the first test, the computer acted as a recommendation engine. It was shown a list of movies a person had watched, described by their titles and genres in normal English, followed by the machine codes for those movies. The computer then had to predict the next movie. Instead of guessing a title, it generated the machine code for the next movie, along with its year and genre. This approach kept the precise structure of the data intact. In the second test, the computer acted as a legal researcher. It was given a paragraph of text from a new court case and asked to find the specific sentence from an old case that supported the argument. Again, the computer did not just search for keywords; it generated the machine code representing the exact paragraph of the old case that matched the context.

The results of these tests were clear and consistent. On the movie recommendation task, the new method significantly outperformed other systems that tried to solve the problem using only text or only codes. On the largest dataset of movies, the new approach improved the accuracy of finding the right recommendation by more than one hundred percent compared to some previous methods. In the legal task, it found the correct supporting sentence far more often than systems designed specifically for legal text. The researchers found that the key to this success was not just adding the codes, but keeping the human language in the mix. When they removed the text descriptions and asked the computer to work only with the codes, the system struggled and lost its way. The human language provided a necessary context, a kind of grounding that helped the computer understand the meaning behind the symbols.

This work suggests a new path for how artificial intelligence might evolve. For a long time, the field has treated language and structured data as separate domains, requiring different tools for each. The researchers demonstrated that a unified framework can learn to speak both languages fluently, treating a machine code and a human word as equal partners in a conversation. This does not mean the computer stops understanding human language; rather, it expands its ability to interact with the digital world directly. The study showed that by integrating these machine-native symbols into the model's vocabulary, the system could handle complex, structured information without losing the rich world knowledge it gained from reading human text. The researchers noted that while the system excelled at these prediction tasks, they did not test whether this new way of thinking affected the model's ability to write poetry or tell stories, leaving that question for future exploration.

The implications of this approach extend beyond just movies and court cases. The researchers showed that the same framework could be applied to different types of structured data without needing to change the underlying architecture of the model. Whether the task was predicting the next item in a sequence or finding a specific passage in a vast legal database, the method remained the same. This flexibility suggests that in the future, artificial intelligence systems might not need to be built from scratch for every new type of data. Instead, a unified framework could be taught to understand the unique "language" of any system, from medical records to financial transactions, by simply learning to generate the right symbols. The study provides a proof of concept that the divide between human language and machine code can be bridged, allowing artificial intelligence to operate more naturally within the structured environments that power much of our modern world.

Technical Summary: UniLang – A Unified Generative Framework for Integrating Machine-Native Symbols

Problem Statement

Modern AI systems frequently represent entities, behaviors, and structured information using discrete machine-native symbols (e.g., vector-quantized codes, semantic IDs, ICD codes) rather than natural language. While these representations are compact, computationally efficient, and preserve explicit structural information, they exist outside the linguistic token space of pretrained Large Language Models (LLMs).

This creates a fundamental methodological divide:

Verbalization Approaches: Converting structured data into natural language descriptions allows LLMs to process them but risks obscuring native structure, introducing ambiguity, and sacrificing machine-level precision.
Task-Specific Models: Operating directly on machine-native symbols preserves structure but fails to leverage the linguistic and world knowledge encoded in pretrained LLMs within a unified generative space.

The core limitation is representational: pretrained LLMs lack a unified interface to treat machine-native symbols as first-class generative units alongside natural-language tokens.

Methodology: The UniLang Framework

UniLang proposes a unified generative framework that extends pretrained LLMs to jointly model natural-language tokens and machine-native symbols under a single autoregressive objective. The framework consists of three primary components:

1. Construction of Machine-Native Representations

UniLang is agnostic to the specific generation mechanism but utilizes Residual Quantized Variational Autoencoders (RQ-VAE) in this work.

Process: Textual descriptions of items (e.g., movie titles, legal contexts) are encoded into dense embeddings, which are then discretized into a sequence of codes across multiple quantization levels ( $l=3$ ).
Disambiguation: An additional level is appended to mitigate code collisions.
Tokenization: Level-specific prefixes are prepended to these codes to form Semantic IDs (SIDs) (e.g., A11, B43). These SIDs, combined with natural language, form an extended vocabulary where machine tokens function as distinct generative units.

2. Vocabulary Expansion and Semantic Grounding

To integrate these new tokens, UniLang performs a contrastive grounding procedure:

Vocabulary Extension: The pretrained LLM's vocabulary is expanded with a fixed set of machine tokens ( $V_{ML}$ ), initialized with learnable embeddings.
Contrastive Alignment: A contrastive loss (InfoNCE style) aligns the embedding of a machine token sequence ( $z^{ML}_i$ ) with the embedding of its corresponding natural-language textual description ( $z^{NL}_i$ ).
Training Constraint: During this phase, only the machine token embeddings are updated; the pretrained LLM parameters and natural-language embeddings remain frozen. This ensures the new symbols are grounded in the model's existing semantic space without catastrophic forgetting.

3. Unified Autoregressive Modeling

Once grounded, the framework treats all tokens (natural language and machine symbols) identically:

Single Objective: The model predicts the next token in a sequence using a standard autoregressive objective.
Task Agnosticism: No task-specific architectural changes are required. Tasks are defined by the construction of input-output sequences.
Structured Generation: Outputs are generated in a field-wise manner using delimiter tokens (e.g., <year>, <genre>, <sid>). For example, in recommendation, the model generates metadata fields followed by the machine-native SID, rather than predicting a raw ID directly.

Key Contributions

Unified Representational Interface: The paper formulates structured prediction as typed autoregressive generation over heterogeneous token types, allowing machine-native symbols to function as first-class units within a single pretrained LLM.
Grounded Integration: It introduces a vocabulary-extension and contrastive grounding procedure that maps machine-native symbols into the LLM's representation space, enabling joint modeling while retaining discrete structure.
Generality Across Domains: The framework is demonstrated on two structurally distinct tasks—sequential recommendation and legal precedent prediction—without requiring task-specific architectural modifications.

Experimental Results

The authors evaluated UniLang on six real-world benchmarks: three sequential recommendation datasets (Amazon Beauty, MovieLens 1M, MovieLens 20M) and three legal precedent prediction splits (LePaRD 10k, 20k, 50k).

Sequential Recommendation

Baselines: Compared against discriminative models (SASRec, BERT4Rec, S3-Rec) and generative baselines (P5, TIGER).
Performance: UniLang consistently outperformed strong baselines. On the MovieLens-20M dataset, it achieved a 114.96% improvement in Recall@5 and 151.73% improvement in NDCG@5 compared to the best baseline (TIGER).
Stability: Statistical analysis across multiple runs confirmed the stability of these gains.

Legal Precedent Prediction

Baselines: Compared against retrieval methods (BM25, fine-tuned SBERT) and classification models (LEGAL-BERT, DistilBERT).
Performance: UniLang significantly outperformed all baselines. On the LePaRD 10k dataset, it achieved a 49.36% improvement in Recall@1 over the best baseline (LEGAL-BERT).
Significance: The results demonstrate that the framework generalizes to fundamentally different problem structures (chronological user interaction vs. semantic passage retrieval) using the same modeling approach.

Ablation Studies

Ablation experiments on MovieLens-20M revealed:

Natural Language Context: Removing natural language fields (NLRemoved) led to performance degradation and overfitting, indicating that natural language provides essential regularizing context.
Structuring: Removing typed delimiters (NoTypeDelim) reduced performance, highlighting the importance of explicit structuring for optimization.
Grounding: Using randomly initialized embeddings without contrastive grounding (NoWarmup) resulted in a complete failure to learn (zero metrics), proving that pre-aligning symbols to the LLM's latent space is critical.

Significance and Claims

The paper claims that UniLang provides a general interface for extending pretrained LLMs beyond language to model heterogeneous machine-native representations.

Unified Backbone: It demonstrates that a unified framework can replace task-specific architectures for diverse structured prediction problems.
Scalability: Unlike systems that require dedicated user tokens or hashing schemes (which consume significant vocabulary space), UniLang can represent user identifiers directly in natural language, offering better scalability.
Efficiency: By treating structured symbols as first-class tokens, the framework avoids the representational ambiguity and information loss associated with verbalizing structured data, while still leveraging the rich world knowledge of pretrained LLMs.

The authors note that while the framework excels at predictive accuracy, the impact of symbolic fine-tuning on the model's natural language generation quality (fluency and coherence) remains an open question for future work.