Microsoft Patent Recovers Full 3D Body Mesh From a Single 2D Photo
One ordinary photo in, a full three-dimensional model of a person (or any jointed object) out. Microsoft has filed a patent describing a system that figures out how a body is posed and shaped from a single 2D image, no depth camera or body scan required.
What Microsoft's single-image 3D body reconstruction does
A surveillance camera captures a pedestrian mid-stride. A fitness app snaps someone doing a squat. Both situations give you the same thing: a flat, two-dimensional picture of a body in motion. Turning that into a proper 3D model, one that captures the exact angle of every limb, has traditionally required special cameras or lots of additional sensors.
Microsoft's patent describes a way to do that reconstruction from a single ordinary photo. The system first identifies key body landmarks, like wrists, knees, and shoulders, then uses those anchor points to estimate how the whole body is oriented in three-dimensional space. From that pose estimate, it generates a complete 3D mesh, essentially a sculpted digital body.
The target here isn't just people. The patent refers to any "articulated object," meaning anything with joints: arms, legs, robotic limbs, animal skeletons. If you can photograph it, the system aims to reconstruct it.
extracting features from the two-dimensional image by: generating a plurality of image patches from the two-dimensional image; generating a plurality of visual tokens and a classification (CLS) token by processing the plurality of image patches through a vision transformer encoder …
Translation: The system chops the flat photo into pieces and processes them through a special neural network.
How the system goes from pixels to a 3D posed mesh
The pipeline has four main stages, each feeding into the next.
1. Image patch encoding: The input photo is sliced into a grid of small patches, which are fed into a vision transformer encoder (a type of AI model that treats image regions the way language models treat words). This produces two outputs: a set of visual tokens (one per patch, carrying local detail) and a single CLS token (a compressed summary of the whole image, analogous to a table of contents).
2. Landmark prediction: A separate decoder generates heatmaps (probability maps showing where each joint is most likely to appear) and extracts initial coordinates and confidence scores from them. Then, crucially, the CLS token is used to refine those coordinates. Because the CLS token has seen the full image context, it can correct landmark guesses that look locally plausible but are globally wrong, for example, a wrist predicted on the wrong side of the body.
3. Pose estimation: The refined landmark positions are used to pool features from the visual tokens (a process called landmark-based feature pooling, meaning it samples the richer image detail only around relevant joints). These per-landmark features pass through transformer decoder blocks to produce pose parameters: numerical values describing every joint angle.
4. Mesh generation: The pose parameters are fed into a parametric model (a pre-built mathematical body template, similar to how a tailor's dress form works) to output the final 3D mesh.
Landmarks are predicted by generating landmark heatmaps, determining initial landmark coordinates and confidence scores, and refining these coordinates using the CLS token.
Translation: It maps out key body points on the image and then fine tunes their exact positions.
What this means for avatars, fitness apps, and AR clothing
For app developers and platform builders, this kind of single-image 3D reconstruction is the missing piece for features like virtual try-on, motion analysis, or realistic avatar creation that work without asking users to own a depth-sensing camera. Your phone camera becomes enough.
Microsoft's run of computer-vision filings suggests the company is building toward richer spatial understanding across its products, from Teams backgrounds to mixed-reality tools. A reliable single-photo body reconstruction would matter most in products where you can't assume specialized hardware: web apps, mobile devices, and any scenario where the input is just a regular video frame captured in the wild.
Microsoft's ninth filing in the AI simulation filings we cover since May follows robot mission pre-testing and real-time game audio remixing.
The CLS-token refinement step is the interesting design choice here, and it comes with a real cost. The system borrows the global image summary to fix local landmark mistakes, which is clever. But that global token is a compression of everything in the frame, including background clutter. In a crowded scene, the CLS token may carry as much noise as signal, which could make the refinement step counterproductive for exactly the messy real-world images where you'd most want it.
The parametric-model step is the other place where the design trades flexibility for practicality. Pre-built body templates work well for average builds and standard poses, but they impose a ceiling: the output mesh can only look as diverse as the template permits. Unusual body proportions, heavy clothing, or extreme poses can all push the real subject outside what the template was designed to represent.
That said, the trade reads as reasonable for a first-pass system. Landmark-anchored pooling is a meaningful structural choice over brute-force global attention, and the two-stage refinement addresses a known failure mode in heatmap-based detectors. The design is thoughtful engineering on a hard problem, not a shortcut.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
7 drawing sheets from US 2026/0260423 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →