💻 computer science

Human-Centric Image Retrieval Using Scene Graphs with Person Layout and Nonverbal Social Cues

This paper proposes a human-centric image retrieval method that leverages scene graphs to encode 2.5D person layouts and nonverbal social cues—such as gaze, orientation, and gestures—enabling the structural comparison of interpersonal interactions beyond global visual similarities.

Original authors: Yeon-Jae Lee, Jong-Hyun Kim

Published 2026-10-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yeon-Jae Lee, Jong-Hyun Kim

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast digital libraries of our time, finding a specific image often feels like searching for a needle in a haystack of pixels. Traditional computer systems have become quite good at recognizing what objects are present in a photo; they can tell you that a picture contains a dog, a tree, or a group of people. They can also match images based on color, lighting, or the general mood of a scene. However, these systems often struggle to understand the story happening between the people in the photo. A computer might see two images with the same number of people and similar backgrounds and assume they are the same, even if the people in one image are facing away from each other in silence, while in the other, they are leaning in, making eye contact, and sharing a conversation. The meaning of a human scene is not just in who is there, but in how they are arranged, how they look at one another, and the invisible lines of attention that connect them.

Researchers at Inha University in South Korea have developed a new way to teach computers to see these subtle social connections. Instead of just looking at the overall picture, their method builds a structural map of the people within the scene. They treat each person as a point on a map and draw lines between them to represent how they are interacting. These lines are not random; they are based on real visual clues like where a person is looking, how their body is turned, the position of their arms, and how close they are standing to one another. By combining these details with a sense of depth—figuring out who is standing in front of whom—the system creates a detailed blueprint of the social situation. This allows the computer to search for images not just by what they look like, but by how the people inside them are relating to each other.

The team's approach begins by taking a photo and identifying every person in it. For each person, the system pinpoints their location and estimates their body shape using key points like the shoulders, elbows, and knees. Crucially, the system does not stop at a flat, two-dimensional view. It uses a technique to estimate how far away each person is from the camera, creating a sense of depth that helps distinguish who is standing in the foreground and who is in the background. This depth information is vital because two people might appear close together on the screen but actually be standing far apart in the real world, or vice versa. Once the positions are mapped, the system analyzes the space between every pair of people to determine the nature of their connection. It checks if their heads are turned toward each other, if their bodies are aligned in a way that suggests conversation, or if their gestures indicate they are reaching out to one another.

All of this information is woven together into a single, compact description of the scene. This description captures the number of people, their layout in space, and the strength of the social bonds between them. When a user wants to find a similar image, the system compares this description against a massive database of millions of photos. It does not simply look for pictures with the same colors or backgrounds. Instead, it looks for scenes where the people are arranged in a similar way and where the invisible lines of interaction match the query. For example, if a user searches for an image of two people having a serious conversation, the system will prioritize images where two people are facing each other with focused attention, even if the background is completely different from the search image. It will ignore images where people are standing near each other but looking in opposite directions, even if the lighting and colors are identical.

The researchers tested their method on a database containing 560,000 images and found that this structural approach significantly improved the quality of the search results. In many cases, the system successfully retrieved images that looked visually different but shared the same social dynamic. When they removed the social connection part of their analysis, the system's ability to distinguish between different types of interactions dropped sharply. For instance, in a test involving 26 different search images, removing the social cues caused the top result to change in 18 cases. This proves that the system was relying on the subtle details of human interaction—like gaze and body orientation—to make its decisions, rather than just matching the general appearance of the photo. The method also showed promise in analyzing video, where it could identify moments when the group dynamic shifted, such as when a crowd suddenly reorganized or when a conversation began, distinguishing these social changes from simple visual shifts like a change in lighting or a camera angle.

While the system is powerful, the researchers are careful to note its boundaries. It relies on the computer's ability to detect people and estimate their poses, which can be difficult in crowded scenes, low light, or when people are partially hidden. The system does not know what the people are saying or what they are thinking; it only interprets the physical signals visible in the image. It sees a body turned toward another person and infers a connection, but it cannot confirm the intent behind that gesture. Despite these limitations, the work represents a significant step forward in how machines understand human-centric scenes. By moving beyond simple object recognition to map the complex web of nonverbal communication, the researchers have given computers a new way to see the world: not just as a collection of objects, but as a series of relationships. This allows for a more intuitive and meaningful search experience, where the answer to a query is found in the story the people are telling, rather than just the colors they are wearing.

Technical Summary: Human-Centric Image Retrieval Using Scene Graphs with Person Layout and Nonverbal Social Cues

Problem Statement

Existing image retrieval methods predominantly rely on global visual features (e.g., color, texture, background) or object-level semantic information. While effective for general scene similarity, these approaches struggle to capture the spatial arrangements and interaction structures critical to human-centric scenes. Specifically, standard methods fail to distinguish between images that share similar backgrounds or object counts but differ significantly in nonverbal social cues, such as interpersonal distance, gaze direction, body orientation, and gestures. For instance, two people standing close together may not be interacting if they face opposite directions, whereas people further apart may be engaged in conversation if their orientations align. The core problem addressed is how to retrieve images based on the structural similarity of human-centric scenes—specifically the layout and nonverbal relational structures—rather than mere global visual appearance.

Methodology

The proposed framework models a human-centric scene as a Scene Graph where people are nodes and nonverbal social cues are edges. The retrieval process involves four main stages:

1. Person Detection and Node Construction

The system detects people and estimates body keypoints (COCO 17-keypoint format) from an input image. Each person $p_i$ is represented as a node $v_i$ containing:

2D Position: A reference point on the image plane (prioritizing ankles, then hips, then bounding box center based on detection confidence).
2.5D Position: A relative depth-aware position computed using monocular depth estimation.
Attributes: Person size, average keypoint confidence, pose embedding, and an in-scene importance score.
Pose Embedding: Generated by a self-supervised trained PoseGCN encoder, converting keypoint structures into a 64-dimensional feature vector.

2. 2.5D Layout Estimation

To address the limitations of 2D-only layouts, the method employs monocular depth estimation (using Depth Anything V2) to compute a representative depth value for each person (median depth within the bounding box). This depth is normalized and combined with image coordinates to create a 2.5D position ( $x^{2.5D}$ ). This representation captures relative front-back relationships without requiring full 3D reconstruction, allowing the system to distinguish between people who appear close in 2D but are at different depths.

3. Nonverbal Social Relation Edge Construction

For every pair of people, a relation edge $e_{ij}$ is constructed based on six nonverbal cues:

Gaze/Head Direction: The degree to which heads face each other.
Body Orientation: Alignment of shoulder/hip vectors.
Arm Gestures: Direction of wrists/elbows relative to the other person.
Proximity: 2.5D Euclidean distance.
Depth Relation: Relative front-back difference.
Importance: A weighted combination of person size, centrality, and confidence.
These cues are normalized and combined into a weighted relation strength score.

4. Retrieval and Scoring

The system uses a two-stage retrieval strategy:

Candidate Shortlisting: The scene graph is summarized into a fixed-length descriptor (380 dimensions) containing statistics on person counts, node attributes, layout distributions, and edge strengths. Cosine similarity on these descriptors rapidly filters the database to the top- $M$ candidates.
Structure-Preserving Matching: For the shortlisted candidates, the system performs Hungarian matching to find optimal node correspondences between the query and candidate graphs.
Final Score Calculation: The final retrieval score $S(I_q, I_c)$ $S (I_{q}, I_{c})$ combines:
- Descriptor Similarity ( $S_{desc}$ ): Global graph similarity.
- Layout Similarity ( $S_{lay}$ ): Absolute position and relative distance structure of matched nodes.
- Social-Relation Similarity ( $S_{soc}$ ): Comparison of relation edge features for matched pairs, focusing on "core" high-strength edges to avoid noise from background figures.
- Person-Count Consistency ( $C_{cnt}$ ): A penalty term that reduces the score if the number of people in the query and candidate differs significantly.

The method is also extended to video analysis via Social Peak detection, identifying time points where the frame-wise social relation structure undergoes significant reorganization, distinct from simple motion or visual changes.

Key Contributions

Scene-Graph Representation: A novel method for human-centric retrieval that jointly represents person layout and nonverbal social cues (gaze, orientation, gestures) as a structured graph, moving beyond global visual embeddings.
2.5D Person Layout: The integration of monocular depth estimation to create a relative 2.5D layout, enabling the system to account for front-back relationships and depth-aware proximity, which 2D layouts cannot capture.
Explicit Nonverbal Modeling: The construction of relation edges that explicitly model the strength and structure of potential social interactions based on gaze, body alignment, and gestures, rather than just object co-occurrence.
Structure-Preserving Retrieval Score: A composite scoring mechanism that balances global descriptor similarity, layout similarity, social-relation similarity, and person-count consistency, allowing for the retrieval of images with similar interaction structures despite visual differences.
Empirical Validation: Qualitative analysis and component removal experiments demonstrating that nonverbal relation cues significantly adjust retrieval rankings for images that are visually similar but structurally different.

Experimental Results

The method was evaluated on a database of 560,000 human-centric images with 26 query images covering conversations, meetings, crowds, and sports.

Qualitative Retrieval: The system successfully retrieved images with similar interaction structures (e.g., face-to-face conversations, group focus) even when backgrounds, lighting, and clothing differed.
Ablation Analysis: Removing the social-relation component ( $S_{soc}$ ) resulted in a reordering of the Top-5 results for 25 out of 26 queries and changed the Rank-1 result for 18 queries. This confirms that nonverbal cues are critical for distinguishing scenes that global features or simple layouts cannot separate.
Video Analysis: In video experiments, "Social Peaks" successfully identified moments of structural reorganization (e.g., players shifting formation, a person leaving a group) that were distinct from low-level visual changes like subtitle overlays or color shifts. The method showed complementary behavior to color and gradient-based change detection.

Significance and Claims

The paper claims that its primary significance lies in transforming relational elements that determine the meaning of human-centered scenes (spatial arrangement, interaction context) into a searchable structural representation.

Complementarity: The authors position this method not as a replacement for global visual retrieval or low-level video change detection, but as a complementary approach. It captures person-to-person relational structures that global embeddings miss and low-level visual cues (color, gradient) cannot interpret.
Structural Understanding: The study demonstrates that effective human-centric retrieval requires modeling how people are arranged and interact, rather than just what objects are present.
Modesty on Intent: The paper explicitly states that the nonverbal cues are structural indicators of observable visual relations (gaze, pose) and do not infer actual psychological intentions, speech content, or specific social identities. The "Social Peak" is defined as a structural transition point, not necessarily a semantic event peak.

The framework provides a foundation for retrieving human-centric scenes based on spatial and relational contexts, offering a more nuanced approach to image and video analysis in scenarios where interaction structure is paramount.