From Pixels to Physicality: Wave-Mediated Interaction above a Screen
This paper presents a transparent visuo-physical panel that utilizes spatiotemporal inverse filtering to focus ultrasound directly from the display surface, enabling dynamic mid-air haptic feedback and object manipulation without obstructing the viewing area.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Most screens we use today are windows that show us a world of light and color, but they stop there. If a video shows a ball bouncing, the ball remains trapped in the glass; if a character blows out a candle, the air in the room stays still. For decades, scientists have tried to bridge this gap, creating displays that can push, pull, or touch the air around them to make digital objects feel real. Some systems use hundreds of small, opaque speakers to blast sound waves into the air, creating invisible pressure points that can lift light objects or be felt by a fingertip. However, these traditional setups often block the view or require bulky equipment placed beside the screen, separating the visual image from the physical sensation.
A team of researchers at the Korea Advanced Institute of Science and Technology has found a way to merge these two worlds into a single, clear sheet of glass. They developed a transparent panel that acts not just as a window for images, but as a giant, invisible speaker that can focus sound waves into specific spots in the air above it. By turning the entire surface of the display into a source of sound, they created a system where visual content and physical interaction happen in the exact same place, without any opaque hardware getting in the way.
The device, which the researchers call a transparent visuo-physical panel, looks like a standard sheet of glass, but it has a ring of tiny electronic components attached to its edges. When the researchers send electrical signals to these edge components, they make the glass vibrate in complex patterns. These vibrations travel across the glass and push against the air, creating focused beams of ultrasound. Unlike a normal speaker that spreads sound in all directions, this panel uses a sophisticated calculation method to steer the sound waves so they meet at a single, precise point in the air, much like a magnifying glass focuses sunlight. Because the glass itself vibrates to create the sound, the entire viewing area remains clear and unobstructed.
To make this work, the researchers had to solve a difficult problem: the sound waves behave unpredictably as they travel through the glass, bouncing and spreading out before they even reach the air. Instead of trying to predict every twist and turn with a simple formula, the team measured exactly how the glass responded to different electrical signals. They then used a computer to work backward, calculating the exact electrical pattern needed to cancel out the messy vibrations and force the sound to focus at a desired spot. This process allows them to create a tight ball of high-pressure sound in mid-air, which can be moved or multiplied as needed.
In their tests, the team demonstrated that this focused sound is strong enough to do real work. They placed a thin strip of paper 50 millimeters above the glass panel. When they focused the sound on the paper, the invisible force lifted it into the air. They showed that they could move this lifting force along a path, picking up a series of paper strips one by one, or create multiple lifting points at the same time to hold several objects simultaneously. The sound pressure they generated reached nearly 1,000 pascals, a level sufficient to physically interact with lightweight objects without touching them.
Beyond moving objects, the researchers explored whether humans could feel these invisible forces. They asked ten volunteers to place their hands above the panel while wearing headphones to block out the sound of the vibration. By turning the sound on and off rapidly, they created a rhythmic pressure that the participants could feel on their skin. The volunteers could not only detect the touch but also tell which direction the sound was moving, such as up, down, left, or right, with an accuracy of over 94 percent. This proved that the panel could provide a form of touch that exists in the air, independent of the screen surface.
The final demonstration brought the visual and physical elements together. The researchers placed the glass panel over a standard computer screen and set up a row of small candles in front of it. On the screen, they played a video of a person blowing. At the exact moment the video showed the person blowing, the panel focused a beam of sound onto the corresponding real candle. The force of the sound was strong enough to extinguish the flame, creating a seamless illusion where the digital action caused a physical result in the real world. This experiment showed that the display could align what you see with what you feel and what happens in the room, all without any opaque devices blocking the view.
The researchers emphasize that this approach is different from previous methods that rely on arrays of hundreds of individual, opaque speakers. Those systems often require the speakers to be placed around the display or use special screens that can obscure the image. By using the glass panel itself as the sound source, this new design keeps the viewing area completely clear. While the current system works best with lightweight objects and requires precise calibration, the work proves that a single transparent surface can serve as both a window for light and a tool for physical manipulation. It opens the possibility for future displays where the boundary between the digital image and the physical world is no longer a barrier, but a shared space where images can reach out and touch the viewer.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.