Samsung Patents Tech That Shows Your Real Keyboard Inside a Virtual World
Typing on a real keyboard while wearing a VR headset is awkward because the headset can't show your hands and keys accurately. Samsung's new patent describes a way to scan your physical keyboard with depth sensors and reconstruct it as a 3D virtual object that lines up precisely with what you actually see.
What Samsung's keyboard-in-VR system actually does
You're wearing a mixed-reality headset and you reach for your keyboard to type, but the image of your hands and keys looks slightly off, misaligned, or blurry compared to real life. That tiny mismatch is enough to make typing feel unnatural and slow you down.
Samsung's patent describes a system that fixes this. The headset's cameras and depth sensors scan the scene, spot the keyboard, and then build a full 3D virtual model of it on the fly. That model is then carefully lined up with the live camera feed so what you see in the headset matches the real keyboard sitting on your desk.
The result is a passthrough view where your keyboard looks solid, correctly shaped, and properly placed in your virtual workspace. You get the feel of typing on real keys without losing immersion in whatever virtual environment you're working in.
… identifying, using at least one processing device of the electronic device, a physical input device captured within the image frames; generating, using the at least one processing device, a three-dimensional (3D) virtual image of the physical input device …
Translation: The headset uses its cameras to spot your real keyboard and creates a digital 3D model of it.
How the depth scan becomes a matched 3D overlay
The patent outlines a four-step pipeline running on the headset's processor.
- Capture: Multiple sensors (cameras plus a depth sensor, similar to the kind in modern smartphones) grab image frames of whatever is in front of the user.
- Identify: The system detects that a physical input device, specifically a keyboard or similar peripheral, appears in those frames. This likely uses object-recognition software trained to spot keyboard shapes.
- Reconstruct: Using the depth data (a per-pixel distance map of the scene), the system generates a 3D virtual image of the keyboard rather than just a flat photo of it.
- Match and render: The 3D model is aligned with a passthrough-transformed image (meaning the live camera feed that has been corrected for lens distortion and latency) to produce a final composited frame that goes to the display.
The matching step is the core innovation. A flat camera image of a keyboard will look slightly warped or delayed compared to a virtual scene. By building a 3D model and snapping it to the passthrough feed, the system keeps the keyboard looking geometrically consistent no matter which angle you view it from.
… matching, using the at least one processing device, the 3D virtual image and a passthrough-transformed image of the physical input device to generate a final image frame for rendering.
Translation: The system blends the digital model with the live camera feed so you can see your keyboard clearly while in virtual reality.
What this means for mixed-reality productivity headsets
Typing is one of the biggest unsolved problems for mixed-reality headsets aimed at productivity. Meta, Apple, and Samsung are all competing to make headsets people will actually use at a desk for hours, and the keyboard is the one physical object most knowledge workers interact with constantly. A headset that makes your keyboard look wrong, feel laggy, or appear to float in space will lose that user fast.
Samsung's approach treats the keyboard as a proper 3D object rather than a flat video cutout, which is a meaningful step toward making passthrough mixed reality feel physically convincing. It fits a broader pattern of AR hardware patents focused on the unglamorous but critical details of how real objects coexist with virtual ones, the kind of Big Tech patent news in the AR space that rarely gets attention but defines whether a product feels finished or frustrating.
The problem this patent attacks is real and underappreciated. Keyboard mismatch in passthrough headsets is not a niche edge case; it is one of the first things a new user notices and one of the fastest ways to break immersion during work. The depth-sensor-plus-3D-reconstruction approach matches the scale of that problem because it addresses the geometric root cause rather than papering over it with image processing tricks. Whether Samsung can run this pipeline fast enough on shipping hardware to avoid perceptible lag is the question the patent leaves open, and that gap between a clever method and a comfortable product is where most XR ideas stall.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
13 drawing sheets from US 2026/0236163 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →