Microsoft Patents a Way to Track Objects in 3D Space Using Fitted Geometric Shapes

By Patentlyze Team · Updated Jul 24, 2026

Microsoft is patenting a computer vision technique that wraps geometric shapes around objects in a scene to figure out exactly where they sit in three-dimensional space, and then keeps that estimate updated over time as things move.

How Microsoft's shape-fitting 3D tracker works in plain English

Imagine a security camera watching a parking lot. To know how far away a car is, or whether it's moving toward the camera, the system needs to understand depth, not just what the car looks like. That's surprisingly hard to do from a flat image.

Microsoft's patent describes a way to solve this by first fitting simple geometric shapes (think boxes or cylinders) around known objects in the scene whose real-world size is already known. That lets the system figure out the scale and depth of the whole image. Once it has that calibration, a machine learning model can then estimate where any object in the scene sits in 3D space.

The system also updates its estimates over time, smoothing out errors frame by frame rather than making a fresh, potentially wrong guess with each new image. The result is a more consistent picture of where things are and where they're headed.

How the model fits shapes and calibrates spatial coordinates

The patent outlines a three-stage pipeline for turning ordinary image data into reliable 3D spatial estimates.

• Shape fitting: The system starts with a set of predefined geometric shapes (boxes, ellipsoids, cylinders) and fits them to objects already visible in the scene whose physical dimensions are known. Think of it like draping a 3D wireframe over a parked car of known length to anchor the image in real-world scale.
• Spatial calibration: The parameters from those fitted shapes (how big, how tilted, how far) are used to calibrate the coordinate system of the entire image. This tells the model how to translate pixel positions into real distances.
• ML-based estimation with temporal integration: A machine learning model then estimates the 3D position and orientation of target objects in the now-calibrated image. Crucially, the patent also describes updating the model's learning rate (how fast it adjusts its estimates) in response to external events, and integrating updates over time so the spatial estimates stay stable across frames rather than flickering.

The temporal integration step is the part that lifts this above a simple single-frame detector. By carrying information forward across multiple images, the system can track objects through a scene more reliably, even when detection is briefly noisy or occluded.

What this means for computer vision and spatial computing

For any application that needs to know where physical objects are in 3D space, including robotics, autonomous vehicles, augmented reality headsets, and industrial computer vision, reliable depth estimation from standard cameras is a long-standing problem. Dedicated depth sensors (LiDAR, structured light) solve it but add cost and bulk. A purely image-based approach that can calibrate itself using known objects in the scene is cheaper to deploy and easier to retrofit onto existing camera hardware.

Microsoft's angle here is plausibly connected to its mixed-reality work (HoloLens) and its Azure cloud computer vision services. A system that tracks objects in 3D without special sensors could feed into both product lines, and the adaptive learning rate detail suggests this is designed to stay accurate in dynamic, real-world environments rather than controlled lab settings.

Editorial commentary on a publicly published patent application. Not legal advice.