TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications

The paper proposes TaiChi, a novel Vision-Language Model framework that enhances multimodal token communications through a dual-visual tokenizer, a Bilateral Attention Network for compact token fusion, and a KAN-based projector for precise cross-modal alignment, ultimately demonstrating superior performance in a joint VLM-channel coding system.

The Gist

Imagine you are trying to send a complex, high-definition photo of a bustling city street to a friend on the other side of the world. In the old days of communication, you would have to send every single pixel of that photo, one by one. If the connection was bad, the image would arrive pixelated, blurry, or missing chunks.

TaiChi is a new, super-smart system designed to fix this. Instead of sending raw pixels, it sends "ideas" (or "tokens") about the image. Think of it like sending a perfectly written story about the photo instead of the photo itself. Your friend's computer then uses that story to instantly "re-imagine" the picture in their mind.

Here is how TaiChi works, broken down into simple, everyday concepts:

1. The Problem: The "Blurry vs. Tiny" Dilemma

Current AI systems that look at pictures have a hard time. They are like a photographer who can only take two types of photos:

• The Wide Shot: Great for seeing the whole city skyline, but you can't see the faces of the people or the texture of the bricks.
• The Zoom Shot: Great for seeing a single flower petal, but you lose the context of where the flower is.

Most AI tries to do both at once and ends up sending a massive amount of data (too many "words" to describe the picture), which clogs up the network and causes errors.

2. The Solution: The "Dual-Lens Camera" (Dual-Visual Tokenizer)

TaiChi solves this by using a two-lens camera system:

• Lens A (The Wide Angle): It looks at a slightly blurry, low-resolution version of the image to understand the big picture (e.g., "It's a busy street with a red bus").
• Lens B (The Macro Lens): It zooms in on the high-resolution details to catch the fine stuff (e.g., "The bus has a scratch on the side," or "The person is wearing a blue hat").

By using both lenses, TaiChi gets the best of both worlds without getting confused.

3. The "Smart Editor" (Bilateral Attention Network)

Now, TaiChi has two streams of information: the "Big Picture" and the "Fine Details." If it just mashed them together, it would be messy.

Enter the Bilateral Attention Network (BAN). Think of this as a super-smart editor sitting at a desk with two stacks of notes.

• The editor looks at the "Big Picture" note and asks, "Where exactly is that red bus? Let me check the 'Fine Details' stack to find the specific pixels."
• Then, the editor looks at the "Fine Details" note and asks, "Is this scratch on the bus important? Let me check the 'Big Picture' stack to see if it fits the story."

The editor cross-references the two, filters out the junk (like background noise), and writes a short, perfect summary that contains all the important details but is much shorter than the original. This saves a huge amount of space.

4. The "Universal Translator" (KAN Projector)

The AI needs to send this summary to a Large Language Model (the "brain" that understands text). But the visual summary is in "Picture Language," and the brain speaks "Text Language."

Old systems used a rigid translator (like a dictionary that only has one definition for every word). TaiChi uses a Kolmogorov-Arnold Network (KAN).

• Imagine a chameleon instead of a dictionary. A chameleon can change its colors to match exactly what it sees.
• The KAN is a translator that can learn and adapt its "voice" on the fly. It doesn't just translate "Red Bus"; it translates the feeling, the context, and the nuance of the red bus so the text brain understands it perfectly, without losing any meaning.

5. The "Noise-Proof Delivery" (Token Communication)

Finally, TaiChi sends these "idea tokens" over the internet.

• Old Way: If a storm hits the internet (noise), you lose pixels, and the photo looks broken.
• TaiChi Way: Because it's sending "ideas" (tokens), if a few words get lost in the storm, the receiving AI is smart enough to guess the missing words based on the context. It's like if you say, "The cat sat on the [blank]," and your friend knows you, they will instantly know you meant "mat" or "sofa" without you needing to say it.

Why is this a big deal?

• Speed: It sends less data, so it's faster.
• Clarity: It understands both the big picture and the tiny details.
• Resilience: It works even when the internet connection is terrible.

In short, TaiChi is like upgrading from sending a fax machine (which breaks easily and sends everything) to sending a smart, context-aware text message that your friend can instantly visualize perfectly, no matter how bad the signal is. It's the future of how we talk to machines and each other in the age of AI.

Technical Summary

1. Problem Statement

The paper addresses the limitations of applying existing Vision-Language Models (VLMs) to Token Communication, a paradigm where multimodal data is transmitted as discrete semantic tokens rather than raw bits. Current VLMs face three critical bottlenecks in this context:

• Limited Token Granularity: Most VLMs rely on single, low-resolution tokenizers that fail to simultaneously capture global context and fine-grained pixel details, preventing adaptive transmission based on task requirements.
• Overlong Visual Token Sequences: High-resolution visual features are often encoded into excessively long token sequences. This causes computational bottlenecks (context length limits), dilutes critical information, and incurs high communication overhead, leading to information loss during transmission.
• Inadequate Cross-Modal Alignment: Traditional projectors use Multi-Layer Perceptrons (MLPs) with fixed activation functions. These struggle to model complex nonlinear relationships between vision and language, often suffering from "spectral bias" (prioritizing low-frequency info) and losing high-frequency visual details during projection.

2. Methodology: The TaiChi Framework

The authors propose TaiChi, a novel VLM framework designed specifically for token communications. It integrates a dual-visual tokenizer, a bilateral attention network, and a Kolmogorov-Arnold Network (KAN) projector.

A. Dual-Visual Tokenizer Architecture

To overcome granularity limitations, TaiChi employs two parallel processing paths:

• Low-Resolution (LR) Tokenizer: Uses a Vision Transformer (ViT) to process downsampled images, extracting robust global conceptual features (the "what").
• High-Resolution (HR) Tokenizer: Uses a Convolutional Neural Network (CNN) to process original high-resolution images, capturing fine-grained local details like textures and edges (the "how" and "where").

B. Bilateral Attention Network (BAN)

To efficiently fuse the global and local feature streams without generating excessive tokens, TaiChi introduces the BAN:

• Mechanism: It utilizes two complementary attention branches:
  1. Top-Down: High-level features query low-level maps to extract relevant fine details.
  2. Bottom-Up: Low-level features query high-level maps to capture global semantic guidance.
• Function: Unlike dense global cross-attention, BAN performs localized spatial correspondence attention. This reduces redundancy, suppresses background noise, and fuses the streams into a compact, high-fidelity set of visual tokens.

C. KAN-based Projector

To replace the standard MLP projector, TaiChi employs a Kolmogorov-Arnold Network (KAN):

• Innovation: KANs incorporate learnable activation functions (combining SiLU and B-splines) directly into the network weights, rather than using fixed activations.
• Benefit: This allows the model to automatically learn optimal nonlinear mappings between visual features and the text semantic space. It effectively mitigates spectral bias, ensuring high-frequency visual details are preserved and accurately aligned with the LLM's semantic space.

D. Instructional Diffusion Refinement (IDR)

For multimodal output (e.g., generating images from text), TaiChi uses IDR. Instead of passing raw LLM text directly to a diffusion model, the LLM internally optimizes the prompt using structured templates to refine instructions (e.g., specifying object counts, spatial arrangements) before generation, bridging the "semantic gap."

3. Token Communication System

TaiChi is integrated into a Multimodal and Multitask Token Communication System:

• Transmitter: Extracts visual tokens via the Dual-Tokenizer and BAN, aligns them via the KAN projector, and encodes them with text tokens using the LLM.
• Channel Coding: A Joint VLM-Channel Coding scheme is proposed. The VLM and channel encoder/decoder are optimized collaboratively. This allows the system to learn token representations that are inherently robust to channel noise (AWGN) and fading (Rayleigh).
• Receiver: Decodes the received symbols back into tokens and reconstructs the semantic output (text or image) using the LLM decoder.

4. Key Contributions

1. Dual-Visual Tokenizer: A novel architecture that decouples global and local feature extraction, enabling flexible, task-adaptive visual representation.
2. Bilateral Attention Network (BAN): A bidirectional fusion mechanism that creates compact, information-rich visual tokens while significantly reducing sequence length.
3. KAN-based Projector: The first application of KANs in VLM projectors to achieve precise, learnable nonlinear alignment, overcoming the limitations of MLPs.
4. Joint Optimization Framework: An end-to-end token communication system that jointly optimizes the VLM and channel coding, demonstrating superior robustness against channel impairments compared to sequential optimization.

5. Experimental Results

The authors evaluated TaiChi on multiple benchmarks and communication scenarios:

• Zero-Shot Performance: TaiChi outperformed state-of-the-art VLMs (including LLaVA-1.5, InstructBLIP, and BLIP-2) on benchmarks like VQAT, MMB, and MME. Notably, TaiChi with a smaller Gemma-2B LLM outperformed larger models, and TaiChi with Qwen-2.5-14B achieved the highest scores.
• Ablation Studies: Removing either the BAN or the KAN projector resulted in significant performance drops, confirming the necessity of both components for feature fusion and alignment.
• Communication Robustness:
  • In VQA and Text Classification tasks over AWGN and Rayleigh fading channels, TaiChi consistently achieved higher accuracy than systems based on MobileVLM or other DeepSC baselines.
  • Joint Coding: Systems using the proposed joint VLM-channel coding scheme significantly outperformed those with separate, sequential optimization, proving that end-to-end training enhances resilience to noise and interference.

6. Significance

This work represents a significant step toward 6G intelligent communications. By shifting from bit-level transmission to token-level semantic communication, TaiChi demonstrates that:

• VLMs can serve as the core engine for next-generation communication systems.
• Architectural innovations (Dual-Tokenizer, BAN, KAN) can solve the fundamental trade-offs between detail preservation, token efficiency, and cross-modal alignment.
• Joint optimization of semantic extraction and channel coding is crucial for maintaining high fidelity in noisy physical environments, paving the way for robust, AI-native communication networks.