Microsoft Patents a Way for Doctors to Highlight Your Body During 3D Video Calls
During a video call with your doctor, the phrase "I'm concerned about your left shoulder" can feel vague and hard to follow. Microsoft is patenting a system that listens to the doctor, finds the body part they just named, and lights it up on a 3D scan of you in real time.
What Microsoft's 3D body-annotation system actually does
Every time a doctor on a video call says "look at this area" and waves vaguely at a screen, the patient is left guessing. That gap between what a clinician means and what a patient understands is exactly what this patent targets.
Microsoft's system works by scanning you during a 3D video call and building a live digital replica of your body. When the doctor speaks, the software picks out the body-part names in the conversation and highlights those exact areas on the digital copy, so both you and the doctor are looking at the same thing at the same time.
So if your doctor says "I want to take a closer look at the right knee and the lower back," those regions glow or change color on the 3D model, leaving no room for confusion. The idea is to bring some of the clarity of an in-person exam into a remote appointment.
… highlighting the region of the subject by modifying how the plurality of input mesh vertices appear in a rendering of the input mesh of the subject.
Translation: The system visually emphasizes a specific body part on the screen during the video call.
How the system maps spoken words to a 3D body model
The patent describes a multi-step pipeline that connects live video capture, 3D body modeling, and speech understanding.
- Mesh capture: Cameras record you during the call and generate an input mesh (a three-dimensional wireframe of your body made from thousands of tiny surface points).
- Pose estimation and parameterized model: The system reads your posture from that mesh and builds a parameterized model (a standard digital body template, like a flexible mannequin, adjusted to match your proportions and pose).
- Vertex mapping: It then creates a lookup table that matches each point on the template body to the corresponding point on your actual mesh. This is what allows a generic label like "left foot" to translate into the precise surface region of your specific scan.
- Speech-to-region identification: As the medical practitioner speaks, the system detects anatomical region names in the conversation and looks those names up in the mapping.
- Highlighting: The matched surface points on your live 3D model are visually modified, changing color or brightness, so both parties can see exactly which area is being discussed.
The claim focuses on the mapping step as the technical heart: bridging a standard anatomical vocabulary to a person-specific 3D scan, in real time.
For example, if the practitioner says “I need to further examine the left foot and the right shoulder”, the display of the input mesh may highlight the left foot and the right shoulder of the subject.
Translation: Doctors can point out specific body parts verbally and the software will automatically light them up.
What this means for remote medical appointments
Remote medical appointments are already common, but they rely on a doctor and patient talking past each other about body parts neither can clearly see or point to. This system tries to fix that by making the conversation spatially concrete. If it works as described, a patient could follow along during a telehealth consultation the same way they would if the doctor were physically present and pointing.
The catch is that it requires 3D scanning hardware, not just a regular webcam. That limits where this could actually show up first: specialized telehealth clinics, hospital systems, or physical therapy platforms that already invest in depth cameras. For ordinary patients dialing in from a laptop, this is not close yet.
Microsoft's 462nd filing in our Microsoft coverage since May adds to a run that includes role-specific meeting assistants and emoji-driven meeting summaries.
Anyone who has tried to describe where it hurts over a video call knows how fast "somewhere around here" collapses into mutual confusion. This would fix that moment.
When your doctor says "I want to look at your left foot and right shoulder," those two spots light up on a 3D model of your actual body, visible to you in real time. You stop wondering whether they meant the outside of the ankle or the heel, and they stop wondering whether you understood them.
The catch is practical: generating that 3D model requires depth-sensing cameras that most people do not own. For now, this lands in specialized clinics rather than your living room, but for patients in those settings, the difference between a clear conversation and a confusing one is not small.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0301936 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in