💻 computer science

JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising

JanusMesh is a fast, training-free framework that generates high-quality 3D visual illusions with distinct semantics from different viewing angles by employing a cross-space dual-branch denoising process for seamless geometric fusion and a view-conditioned texture synthesis module for realistic rendering.

Original authors: Siang-Ling Zhang, Huai-Hsun Cheng, Tsung-Ju Yang, Yu-Lun Liu

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siang-Ling Zhang, Huai-Hsun Cheng, Tsung-Ju Yang, Yu-Lun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical sculpture that looks like a completely different object depending on which side you stand on. If you look at it from the front, it's a peacock. If you walk around to the back, it's a pineapple. But if you look at it from the side, it just looks like a weird, abstract blob.

This is the core idea behind JanusMesh, a new computer program that creates these "3D visual illusions" automatically and very quickly.

Here is a simple breakdown of how it works, using everyday analogies:

The Problem: The Old Ways Were Slow or Messy

Before this paper, making these illusions was like trying to build a house by hand:

The "Slow Sculptor" method: Some programs tried to slowly chip away at a digital block of clay, adjusting it over and over until it looked right from two angles. This took about 40 minutes per object and often resulted in colors that were too bright and garish (like a neon sign that hurts your eyes).
The "Glue" method: Others tried to take a 3D model of a peacock and a 3D model of a pineapple, cut them in half, and glue them together. This looked terrible. You could see the ugly "seam" where they were joined, and the back of the pineapple would peek out when you were supposed to be looking at the peacock.

The Solution: JanusMesh (The "Fast & Smart" Method)

The authors created a system that does this in just 3 to 5 minutes without needing to be trained on specific data first. They call it "Zero-Shot," which means you can just type in any two words (like "a boat" and "a parrot"), and it figures it out immediately.

They do this in two main stages:

Stage 1: The "Shape Shifter" (Geometry)

Think of this as the skeleton of the illusion.

Two Brains, One Body: The computer starts with two separate "dreams" (one for the peacock, one for the pineapple). Instead of keeping them separate, it mixes them together while it is still dreaming.
The Smoothie Blender: Imagine you have a digital 3D shape of a peacock and a pineapple. If you just mash them together, it's a mess. JanusMesh uses a special math trick called SDF Blending. Think of this like making a smoothie: instead of just throwing the fruit chunks together, it blends the distance between the fruit and the air. This creates a smooth, seamless transition where the two shapes merge into a single, unified object that doesn't have any ugly cracks or seams.
The Dance Partner Search: Sometimes, a peacock and a pineapple don't fit well if they are facing the wrong way. The system uses a "search engine" (CLIP) to rotate the objects until their silhouettes match up perfectly before they are blended. It's like finding the exact angle where two puzzle pieces click together naturally.

Stage 2: The "Painter" (Texture)

Once the weird, blended shape is built, it looks like a gray, abstract lump. Now, the computer needs to paint it.

The Magic Projector: Instead of painting the whole object with one color, the system acts like a smart projector. When it looks at the object from the "Peacock angle," it projects a peacock texture onto that side. When it looks from the "Pineapple angle," it projects a pineapple texture.
Seamless Blending: It blends these painted images so smoothly that when you walk around the object, the texture changes naturally without any visible lines where the paint switches.

Why Is This Special?

Speed: It's incredibly fast (3–5 minutes) compared to the old 40-minute methods.
No Training: You don't need to feed it thousands of examples of peacocks and pineapples to teach it how to do this. It just knows how to do it instantly.
Scalability: The paper shows it can even handle three objects at once (like a grape, a pineapple, and bamboo) by rotating them 120 degrees apart, creating a single object that changes three times as you walk around it.

The Bottom Line

JanusMesh is like a digital illusionist that can instantly sculpt a single object that hides two (or three) different identities inside it. It solves the problem of "ugly seams" and "slow processing" by blending the shapes mathematically before they are even fully formed, and then painting them with a view-specific magic that makes the illusion perfect.

Technical Summary: JanusMesh

Problem Statement

The paper addresses the challenge of 3D visual illusion generation: creating a single 3D mesh that reveals entirely different semantic meanings (e.g., a peacock vs. a pineapple) when viewed from specific camera angles, while appearing as an unrecognizable abstract geometry from arbitrary viewpoints.

Existing approaches suffer from significant limitations:

Optimization-based methods (e.g., SDS): These are computationally expensive (taking ~40 minutes per shape) and often produce results with severe color over-saturation and geometric leakage.
Naive stitching (Direct Concatenation): Simply stitching two separately generated objects results in unnatural geometric seams, visible backside leakage, and a failure to maintain geometric coherence, breaking the perceptual illusion.
Single-generation models: Standard text-to-3D models fail to produce distinct, view-dependent semantics, often collapsing into generic shapes or blending both semantics uniformly across all views.

Methodology: JanusMesh

JanusMesh is a fast, zero-shot, and training-free framework that generates dual-semantic 3D illusions in 3–5 minutes. The pipeline consists of two distinct stages:

Stage 1: Dual-Branch Geometry Generation

This stage focuses on creating a geometrically coherent mesh that fuses two distinct semantic shapes.

Dual-Branch Denoising: Building on the TRELLIS architecture (a Rectified Flow 3D generator), the method initiates two independent denoising branches from a shared noise latent, conditioned on two different text prompts ( $y_1, y_2$ ).
Cross-Space Denoising: To avoid distorting the latent distribution, the process decodes latents into voxel space at each denoising step.
CLIP-Guided Orientation Search: Before fusion, the system automatically determines the optimal relative rotation between the two objects. It renders candidate views, uses CLIP to find the view of Object 1 with the highest text-image similarity, and then searches for the rotation of Object 2 that maximizes image-image similarity to Object 1's anchor view. This ensures silhouette alignment prior to blending.
SDF Blending: Instead of averaging binary occupancy grids (which causes geometric discontinuities), the method converts voxel representations into Signed Distance Fields (SDFs). It computes the element-wise average of the SDFs and binarizes the result to create a smooth, intermediate geometry. This blended voxel is re-encoded into the latent space for the next denoising step.
Noise Guidance: To handle geometric conflicts, the framework injects spatial priors via Noise Blending Guidance (mixing a pre-generated guidance latent with noise) or Space Control Guidance (interpolating the guidance latent with noise at a specific timestep $t_0$ ). This steers the generation toward the target semantics while maintaining structural integrity.

Stage 2: View-Conditioned Texture Synthesis

Since the fused geometry from Stage 1 may contain unnatural structures that confuse standard texturing models, a separate texturing stage is employed.

View-Aware Prediction: Using Stable Diffusion with a depth-conditioned ControlNet, the system predicts clean textures for the mesh from specific viewpoints ( $\theta_1$ and $\theta_2$ ).
Mesh Texture Aggregation: The predicted 2D textures are un-projected onto the 3D surface. A cosine-weighted blending strategy aggregates these multi-view contributions based on surface normals, ensuring seamless transitions and assigning distinct textures to specific angular sectors corresponding to the target prompts.

Extension to Three Objects

The framework naturally scales to three-object illusions by adding a third denoising branch and fixing target angles at $0^\circ, 120^\circ,$ and $240^\circ$ . Due to increased geometric complexity, Space Control Guidance is mandatory to enforce stronger structural constraints.

Key Contributions

Zero-Shot 3D Illusion Framework: The first method to extend generative multi-view illusions from 2D to fully textured 3D meshes without per-shape optimization.
Two-Stage Architecture: A novel pipeline featuring dual-branch denoising with SDF blending and CLIP-guided alignment for geometric integrity, coupled with view-conditioned texturing for semantic coherency.
Comprehensive Evaluation: A rigorous evaluation protocol utilizing CLIP similarity, GPT-4.1-mini accuracy, FID/KID, Object Detection metrics, and a novel Boundary Seam Score (Impact Factor) to quantify geometric smoothness.

Experimental Results

Evaluated on 60 distinct object pairs across five categories (birds, mammals, reptiles, plants, artifacts) on a single NVIDIA RTX 4090:

Efficiency: JanusMesh generates results in 3–5 minutes, significantly outperforming SDS-based baselines (~40 minutes) and matching the speed of naive concatenation.
Semantic Recognizability: Achieved 84% GPT Accuracy in identifying target semantics, outperforming Direct Concatenation (76%) and Shape from Semantics (70%).
Geometric Integrity: Achieved the lowest Object Detection Multi-Object Rate (18%) compared to Direct Concatenation (56%), indicating the mesh is perceived as a single unified object rather than two stitched parts.
Visual Quality: Produced the lowest FID (185.555) among competitive methods, indicating high visual realism.
User Study: 78.5% of participants rated the generated semantics as "clearly recognizable," and 71% preferred JanusMesh over all baselines. 91% preferred the CLIP-guided orientation strategy over fixed angles.

Significance and Claims

The paper claims that JanusMesh successfully bridges the gap between 2D optical illusions and true 3D geometry. By decoupling geometry generation and texture synthesis, and employing cross-space denoising with SDF blending, the method overcomes the trade-off between geometric coherence and semantic distinctiveness.

The authors emphasize that their approach is training-free and efficient, making high-quality 3D visual illusions accessible without the computational burden of iterative optimization. While the method inherits some failure cases from the underlying TRELLIS model (e.g., specific animal categories) and currently relies on fixed angles for three-object illusions due to CLIP alignment ambiguities, it establishes a new baseline for fast, zero-shot 3D illusion generation.