🤖 AI

Giraffe: A Mapping Architecture from Hidden Text Representations to Visual Embeddings for Efficient Graphic Design

The paper proposes "Giraffe," a lightweight architecture that efficiently maps hidden text representations to visual embeddings using a single token per image, thereby overcoming the input length limitations of existing methods for complex graphic design generation tasks.

Original authors: Nejla Ghaboosi

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nejla Ghaboosi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, a major divide has long existed between machines that can see and machines that can speak. For years, the most advanced systems could look at a photograph and describe it in words, or read a sentence and answer questions about it. They were brilliant at understanding, but they were not very good at creating. When these systems tried to generate new images, they often stumbled, producing results that were disjointed or lacked the cohesive feel of human design. The challenge became even steeper when researchers tried to build tools that could create entire graphic layouts—posters, flyers, or social media posts—that seamlessly blend text, multiple pictures, shapes, and colors into a single, harmonious composition. The problem was not a lack of ideas, but a bottleneck in how the computer processed them. To describe a complex image in a way a computer could understand, the system traditionally had to break the image down into hundreds or even thousands of tiny pieces, like a mosaic made of thousands of individual tiles. This made the computer's "thought process" incredibly long and slow, often causing it to lose track of the overall picture before it could finish the job.

A researcher at Canva Research has proposed a new way to solve this problem, introducing an architecture they call Giraffe. The core idea is simple yet transformative: instead of forcing the computer to describe an image using a long string of thousands of tiny tokens, the system learns to represent an entire image with just a single, special marker. Imagine a library where every book used to be described by a thousand-page summary, but now, a single, unique code on the spine is enough to tell the librarian exactly what is inside. This shift allows the artificial intelligence to handle long, complex sequences of text and images without getting overwhelmed. By compressing the visual information of a whole picture into one compact unit, the model can focus its attention on how that picture fits with the surrounding text and other design elements, rather than getting bogged down in the details of the picture itself.

The researcher built a specific mapping system to make this compression work. They designed a structure that acts like a translator, taking the hidden, abstract thoughts of the language model and converting them into the specific language that visual models understand. This translator is built from two distinct parts that work together during the learning phase. One part is the main worker, responsible for doing the actual conversion of the single image marker into a visual representation. The second part is an assistant that helps the main worker learn the task more effectively. During the training process, the assistant analyzes the visual data and helps the main worker understand the relationship between the text and the image. However, once the system is trained and ready for real-world use, the assistant is removed. This leaves behind a lightweight, efficient tool that can generate complex designs quickly, without the extra weight of the training partner. The system was tested on a massive dataset of over 1.8 million professional graphic designs, ranging from business cards to social media banners, teaching it how to arrange text, colors, and images in a way that looks natural and appealing.

When the researcher tested this new approach against older methods, the difference was clear. The traditional systems, which relied on describing images with long lists of text, often produced designs that felt cluttered or disjointed. The images they generated frequently clashed with the text or each other, lacking a unified style or color theme. In contrast, the Giraffe architecture produced designs that were visually coherent and stylistically consistent. When asked to create a poster with multiple images, the new model could arrange them so that they shared a common color palette and mood, creating a harmonious whole. The system demonstrated that by using just one marker per image, it could capture the essence of the picture—its style, its subject, and its colors—without needing to spell out every detail. This allowed the model to generate designs that were not only more complex but also more aesthetically pleasing, successfully blending text and visuals into a single, seamless output.

The study also explored whether this method could work in reverse, taking an existing image and turning it into a full design layout. In these tests, the system was shown a reference image and asked to generate a graphic design that mirrored its style and content. The results showed that the model could closely match the original image's semantic meaning and visual style, confirming that the single marker successfully carried the necessary information. The researcher found that the system did not need to be retrained from scratch to handle these visual tasks; it could leverage the knowledge it had already learned about language to understand the visual markers. This suggests that the ability to compress visual data into a single token is a powerful tool that can be applied to various media, potentially extending beyond static images to video and audio in the future.

Ultimately, the Giraffe architecture represents a significant step forward in how artificial intelligence handles the creation of visual content. By solving the problem of how to efficiently represent images within a language model, the researcher has opened the door to generating complex, multi-element designs that were previously too difficult for these systems to handle. The work demonstrates that the key to better design generation may not be in adding more complexity to the description of an image, but in finding a simpler, more efficient way to represent it. As these systems continue to evolve, they promise to make the creation of professional-quality graphic design more accessible, allowing computers to act not just as observers of the visual world, but as true collaborators in bringing creative ideas to life.

Technical Summary: Giraffe – A Mapping Architecture for Efficient Graphic Design

Problem Statement

Multimodal Large Language Models (MLLMs) have advanced significantly in visual comprehension tasks (e.g., VQA, captioning) but remain limited in their ability to generate visual content. Existing approaches to bridge this gap generally fall into two categories:

Retrieval-based: Mapping text queries to existing image libraries (e.g., FROMAGe), which fails when specific styles or subjects are unavailable in the library.
Generation-based: Translating hidden token representations into visual embeddings or raw image data (e.g., GILL, Emu, Chameleon).

A critical limitation in current generation-based methods is the representation of images. Most approaches utilize multiple specialized tokens (e.g., 16 to 1024 tokens) to represent a single image. In graphic design tasks, where outputs require a seamless blend of thousands of tokens (text, multiple images, SVG shapes, layout), this multi-token representation drastically increases sequence length. This leads to two major issues:

Capacity Constraints: The sequence often approaches or exceeds the model's maximum token capacity.
Attention Limitations: The limited attention span of transformer architectures struggles to capture long-range dependencies across such extended sequences, resulting in incoherent or stylistically inconsistent designs.

Furthermore, methods relying on textual descriptions to represent images (as a baseline) fail to accurately capture style, color themes, and semantic nuances, leading to designs with poor visual harmony.

Methodology: The Giraffe Architecture

The paper proposes Giraffe, a novel architecture designed to map hidden token representations from a language model to the embedding space of a visual model (specifically CLIP ViT-L/14) using a single [IMG] token per image. This approach allows the language model to integrate text and image tokens within long sequences without exceeding capacity or losing coherence.

Architecture Overview

The Giraffe architecture consists of two shallow Multi-Layer Perceptron (MLP) blocks arranged in an L-shaped structure:

Main Mapping Block: Responsible for mapping the hidden representation of the [IMG] token from the language model to the normalized visual embedding space.
Assisting Block: An auxiliary block that aids the main block during training but is omitted during inference to ensure a lightweight solution.

Key Components:

Compression Modules: Each block has a unique compression module ( $f_{\theta1}$ for the main block, $f''_{\theta2}$ for the assisting block).
Shared Expansion Module: Both blocks share a common expansion module ( $f'_{\theta s}$ ) that links them.
Activation Functions: SiLU is used throughout both modules, with the final layer of the expansion module utilizing Tanh.

Training Objectives

The architecture is trained using six distinct loss functions to ensure accurate mapping and alignment:

Mean Squared Error (MSE) Loss ( $L_{b1-MSE}$ ): Minimizes the error between the predicted normalized embedding and the ground truth.
Cosine Similarity Loss ( $L_{b1-COS}$ ): Promotes directional alignment between predicted and ground-truth embeddings in the projection space.
Assisting Block MSE ( $L_{b2-MSE}$ ): Trains the assisting block to approximate the normalized visual embedding from the ground truth (acting as an autoencoder).
Assisting Block Cosine ( $L_{b2-COS}$ ): Ensures directional alignment for the assisting block.
InfoNCE Losses ( $L_{NCE1}, L_{NCE2}$ ): Applied to the bottlenecks of both MLP blocks to further assist the main block in learning the transformation by contrasting positive and negative pairs.
Cross-Entropy Loss ( $L_{CE}$ ): Standard next-token prediction loss for the language model.

The total loss is a weighted sum of these components.

Inference Process

During inference, the assisting block ( $f''_{\theta2}$ ) is removed. The process involves:

The language model generates a sequence of tokens, including [IMG] tokens.
The hidden representation of each [IMG] token is passed through the main mapping block ( $f'_{\theta s} \circ f_{\theta1}$ ) to produce a normalized visual embedding.
This embedding is denormalized and passed directly to a pre-trained image generation model (FLUX with CLIP ViT-L/14 IP Adapter) to generate the image.
No fine-tuning of the diffusion model is required, as the generated embeddings follow a known distribution.

Key Contributions

Single-Token Image Representation: The paper introduces a method to represent an entire image using a single [IMG] token, significantly reducing sequence length compared to multi-token approaches.
Dual-Block Mapping Architecture: The Giraffe architecture utilizes a main block and an assisting block with a shared expansion module. This design facilitates effective learning of the complex mapping transformation while maintaining a lightweight inference footprint.
End-to-End Graphic Design Generation: The system enables the generation of complex graphic designs containing text, layout, and multiple images in a single coherent sequence, addressing the limitations of previous retrieval-only or text-description-based generation methods.

Experimental Results

The model was evaluated on a proprietary dataset of 1.8 million professionally created graphic designs across two tasks:

1. Text-to-Design Generation

Setup: Used Gemma3 4B as the language model. Compared against a baseline where images were represented by textual descriptions passed to FLUX.
Quantitative: The proposed method achieved a FID score of 66.85, significantly lower (better) than the baseline's 81.61, indicating higher quality and diversity.
Qualitative: The proposed method generated designs with greater layout diversity, stronger stylistic coherence, and better color harmony. The baseline often produced designs with minimal variation and disjointed images, whereas Giraffe successfully integrated multiple images that complemented each other.

2. Image-to-Design Generation

Setup: Replaced the language model with a custom encoder-decoder transformer. The task involved generating a design sequence based on a reference image.
Quantitative: Achieved a CLIP cosine similarity of 0.85 ± 0.05 between the input image and the generated design, demonstrating strong adherence to the source material.
Qualitative: The model successfully captured semantic, stylistic, and color information from the input image using a single [IMG] token per image, producing outputs that closely mirrored the reference.

Significance and Claims

The paper claims that Giraffe effectively solves the challenge of generating complex, multi-modal graphic designs by overcoming the token-length and attention-span limitations of current MLLMs. By compressing image information into a single token, the architecture allows for:

Longer Sequences: The ability to handle designs with multiple images and extensive text without hitting capacity limits.
Improved Coherence: The preservation of style, color, and thematic information that is often lost when images are represented solely by text or fragmented tokens.
Efficiency: A lightweight inference solution that leverages pre-trained visual models without requiring fine-tuning of the image generation backbone.

The author concludes that the single-token approach is crucial for achieving higher diversity and complexity in generated graphic designs, enabling language models to produce visually coherent and stylistically consistent outputs that were previously difficult to achieve.