💻 computer science

PointT2I: LLM-based text-to-image generation via keypoints

PointT2I is a novel, fine-tuning-free framework that leverages a large language model to directly generate human pose keypoints from text prompts, which are then used to guide image generation and refined by an LLM-based feedback system for accurate semantic alignment.

Original authors: Taekyung Lee, Donggyu Lee, Myungjoo Kang

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taekyung Lee, Donggyu Lee, Myungjoo Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to give a very specific instruction to a talented but slightly literal-minded artist. You say, "Draw a person doing the 'Boat Pose' in yoga."

A standard AI artist (like the ones currently available) might hear "Boat" and draw a literal wooden boat on water, or it might draw a person sitting in a generic chair, completely missing the specific yoga shape you wanted. It struggles to translate complex body descriptions into the exact physical structure of a human skeleton.

PointT2I is a new framework designed to fix this problem. Think of it as a three-step translator that sits between your words and the final picture, ensuring the artist draws exactly what you described.

Here is how it works, using simple analogies:

1. The "Skeleton Translator" (Keypoint Generation)

Instead of just handing the artist your text prompt, PointT2I first asks a super-smart language expert (a Large Language Model, or LLM) to read your description and build a 3D skeleton in its mind.

The Analogy: Imagine you describe a yoga pose to a robot. The robot doesn't just "guess" the pose; it calculates the exact 3D coordinates of the nose, elbows, knees, and feet. It figures out, "Okay, the feet are on the ground (z=0), the legs are in a 'V' shape, and the torso is leaning back."
Why it matters: This happens without the robot needing to be retrained on yoga videos. It uses its general knowledge of language and human bodies to build the skeleton from scratch.

2. The "Blueprint" (Image Generation)

Once the 3D skeleton is built, PointT2I flattens it into a 2D "blueprint" (a stick-figure map) and hands this to the image generator (the artist).

The Analogy: You are now giving the artist two things: your original text ("Draw a person in the Boat Pose") AND a stick-figure drawing showing exactly where the limbs should go.
The Result: The artist (using tools like GLIGEN or HumanSD) follows the stick-figure blueprint. Because the blueprint is so precise, the artist can't accidentally draw a boat or a sitting person; they are forced to draw the specific yoga pose you asked for.

3. The "Editor" (Feedback System)

This is the clever part. PointT2I doesn't just stop after one try. It has a built-in editor (another LLM) that acts like a quality control inspector.

The Analogy: Imagine the artist draws the picture. The inspector looks at the text, the stick-figure blueprint, and the final drawing.
- If the inspector sees the legs are bent the wrong way, it tells the skeleton builder, "Hey, the blueprint is wrong. Fix the coordinates."
- If the skeleton is right but the drawing looks weird, it tells the artist, "The pose is right, but the image doesn't match the prompt. Try again."
The Loop: This happens in a cycle. The system keeps refining the skeleton and the image until they perfectly match your description.

What Makes This Special?

No "Schooling" Required: Most AI models need to be trained on thousands of yoga photos to learn how to draw them. PointT2I doesn't need that. It uses the language model's existing brainpower to figure out the pose on the fly.
Handles the Weird Stuff: It works great on common poses (like standing) but also on tricky, complex ones (like acrobatics or specific yoga moves) that usually confuse AI.
Flexible Language: It understands if you say "The Boat Pose" (the name) or "A person balancing on their butt with legs in a V" (a description). It translates both into the same perfect skeleton.

Where It Struggles (The Paper's Limitations)

The paper is honest about where the system hits a wall:

Complex Interactions: If you ask for "juggling while doing a handstand," the system might get the handstand right but fail at the juggling. It's still learning how to handle multiple complex actions at once.
Crowds: It works best with one person. If you ask for "a woman leaning on a man's shoulder," it gets the leaning right, but it sometimes misses the subtle detail of them holding hands.
Non-Humans: If you ask for a lion doing a pose, the system gets confused. It tries to make the lion stand on two legs like a human because its "skeleton translator" is trained on human bodies.

In short, PointT2I is a bridge that turns vague words into precise body structures, allowing AI to draw humans in specific poses with much higher accuracy than before, all without needing to be retrained on new data.

Technical Summary: PointT2I

Problem Statement

Text-to-image (T2I) generation models, particularly diffusion-based architectures like Stable Diffusion (SD) and SDXL, have achieved significant progress in generating high-quality images aligned with natural language prompts. However, these models struggle with fine-grained control over complex human poses. While existing methods like ControlNet, T2I-Adapter, GLIGEN, and HumanSD improve controllability by accepting auxiliary inputs (e.g., keypoints, skeleton maps), they inherently rely on external data sources to provide these structural conditions. This reliance creates a bottleneck: if the external pose data is unavailable or misaligned with the specific semantic intent of a text prompt, the generation fails. Furthermore, current approaches often lack the ability to interpret complex, descriptive pose instructions (e.g., "performing a boat pose" or "holding a V-shaped pose") directly from text without prior pose-specific fine-tuning.

Methodology

The authors propose PointT2I, a framework designed to bridge the gap between high-level textual descriptions and structural pose guidance without requiring pose-specific training or external auxiliary data. PointT2I operates as a semantic-to-structural translator, leveraging Large Language Models (LLMs) to infer human pose keypoints directly from text prompts. The framework consists of three sequential components:

1. Keypoint Generation

Instead of relying on external pose datasets, PointT2I utilizes an LLM (specifically ChatGPT o1-preview in experiments) to interpret the input prompt and directly generate 3D human pose keypoints.

Process: The LLM analyzes the prompt to extract pose-relevant semantics (body configuration, ground contact cues) while filtering out irrelevant context.
Output: It generates 3D coordinates for 17 major human body joints.
Constraints: To ensure anatomical plausibility, the generator enforces constraints such as anchoring contact points (feet, hands) to the ground plane ( $z=0$ ) and maintaining realistic body proportions.
Projection: The generated 3D keypoints are projected onto a 2D image plane using an orthographic camera model. The system selects the projection view (from eight possible directions) that maximizes 2D coordinate variance to minimize joint overlap and best preserve spatial structure.

2. Image Generation

The projected 2D keypoints serve as structural guidance (pose guidance) for a pose-aware T2I backbone.

Backbones: The framework is compatible with various pose-conditioned models, including GLIGEN and HumanSD (built on SD architecture), as well as ControlNet and T2I-Adapter.
Mechanism: The image generator conditions on both the original text prompt and the LLM-derived pose guidance. This allows the backbone to render the specific pose described in the text, unlocking its latent structural rendering capabilities without the model needing to interpret the pose semantics directly from language.

3. LLM-Based Feedback System

To refine the output and ensure semantic consistency, PointT2I employs a two-module feedback loop powered by LLMs:

Module 1 (Keypoint Refinement): Evaluates the generated 3D keypoints against the original prompt. If inconsistencies (e.g., incorrect joint positions) are detected, the LLM provides an explanation and regenerates the keypoints.
Module 2 (Image Refinement): Evaluates the generated image against the prompt and the keypoints. If the image misaligns with the intended pose, the system triggers image regeneration.
Iteration: This process iterates (typically up to three rounds) to progressively improve alignment between the text, the structural guidance, and the final visual output.

Key Contributions

The paper identifies three primary contributions:

First LLM-Driven Keypoint Generation: PointT2I is the first framework to leverage LLMs to generate human pose keypoints directly from textual prompts, acting as a semantic-to-structural bridge. This enables pose-aware T2I backbones to generate accurate images from text alone, without any pose-specific fine-tuning or external pose annotations.
LLM-Based Feedback System: The authors introduce a two-module feedback mechanism that refines both the predicted keypoints and the generated images. This system enhances pose-level consistency and alignment with the textual description without relying on ground-truth pose labels or human supervision.
Robustness to Prompt Variations: The framework demonstrates strong robustness across diverse prompt formulations, effectively interpreting both explicit pose names (e.g., "Tree pose") and descriptive instructions (e.g., "standing on one leg with the other foot against the inner thigh").

Experimental Results

The authors evaluated PointT2I on a diverse set of human poses, including common actions, yoga poses, and acrobatic maneuvers, comparing performance against baseline T2I models (SD 1.4 and SDXL).

Quantitative Performance:
- VQAScore & CLIPScore: PointT2I significantly outperformed baselines in VQAScore (which measures compositional reasoning and action fidelity) and CLIPScore across all categories. For example, in the "Boat pose" category, PointT2I achieved a VQAScore of 0.816 (HumanSD) and 0.669 (GLIGEN), whereas SD 1.4 and SDXL scored near zero (0.001 and 0.001, respectively).
- Classification Accuracy: Using a ResNet classifier fine-tuned on the Yoga-82 dataset, PointT2I achieved significantly higher classification accuracy (e.g., 93.9% average for HumanSD vs. 40.0% for SD 1.4), indicating that the generated images are more distinguishable and accurate representations of specific yoga poses.
Qualitative Results: Visual comparisons showed that baselines often failed to capture correct body structures or misinterpreted prompts (e.g., generating a literal boat instead of a yoga pose). PointT2I successfully preserved pose fidelity, accurately rendering complex configurations like splits, handstands, and yoga postures.
Ablation Studies:
- Feedback System: Increasing feedback rounds consistently improved VQAScore and CLIPScore, validating the necessity of the iterative refinement process.
- Prompt Robustness: The framework maintained high performance across original and rephrased prompts, demonstrating insensitivity to specific wording.
- Compatibility: The method generalized well across different image generators (ControlNet, T2I-Adapter, HumanSD, GLIGEN) and different LLMs (ChatGPT o3, Gemini 2.5 pro).
- Template Sensitivity: Removing the "grounding condition" (defining contact with the ground) led to the most significant performance drop, highlighting its critical role in generating plausible poses.

Significance and Claims

The paper claims that PointT2I represents a novel approach to controllable text-to-image generation by decoupling the need for external pose data from the generation process. By using an LLM to translate semantic text into structural keypoints, the framework unlocks the potential of existing pose-aware backbones, allowing them to generate accurate, pose-conditioned images solely from text.

The authors emphasize that this is achieved without any pose-specific training or adaptation, relying instead on a frozen LLM and pre-trained T2I backbones. The work highlights the potential of LLMs to serve as semantic interpreters that can guide diffusion models in complex structural tasks. While the framework shows promise, the authors modestly acknowledge limitations, including challenges with multi-person interactions, complex composite actions (e.g., juggling while handstanding), and the generation of non-human poses, which remain areas for future research. Additionally, the iterative feedback process introduces computational overhead, presenting a trade-off between generation quality and efficiency.