Get the newsletter
Apple · AR / VR · Filed Jun 16, 2026 · USPTO published Oct 1, 2026 · Oct 1 release

Apple Patents a Way to Track Objects in 3D Even When Depth Data Goes Missing

Depth sensors don't always fire on every frame, and when they miss, most 3D tracking systems fall apart. Apple's new patent describes a way to keep the estimate going anyway, using what the camera already knows.

A person uses a tablet to view an augmented reality overlay, while another person wears a headset in a room. Drawing from patent filing US 2026/0301215 A1.
A person uses a tablet to view an augmented reality overlay, while another person wears a headset in a room.All 7 drawings ↓
Publication number
US 2026/0301215 A1(US20260301215A1)
Applicant
Apple Inc.
Filed · Published
Jun 16, 2026 · Oct 1, 2026
Inventors
Justin A. Meiners, Brendan A. McCarthy, Gregory D. Chu, Joshua R. Ford, Pavel V. Dudrenov, Edwin Iskandar
Status when we published
Waiting for an examiner
Document
20 claims
US classification
382/103
Parent application
Continuation of 18/743,459 (filed Jun 14, 2024)(18743459)

How Apple keeps 3D object tracking alive without a depth sensor

Ever tried to measure a piece of furniture with your phone camera, only for the AR overlay to wobble or snap out of place? That's often because your phone's depth sensor didn't catch every frame, leaving the software guessing.

Apple's patent describes a system that handles those gaps more gracefully. When the depth sensor does fire, the system locks in a solid estimate of how big an object is and where it sits in 3D space. When depth data is missing in a later frame, the system leans on that previous size estimate and the regular camera image to keep tracking the object's position. No gap, no wobble.

The practical effect is that 3D measurements and AR overlays stay stable even as your device moves around a room and the sensors skip frames. For everyday tasks like measuring a couch before you buy it or placing a virtual object in your living room, this kind of consistency is what makes the experience feel reliable rather than frustrating.

From the filing · THE ABSTRACT
For each frame of the sequence of frames that includes image data and excludes depth data, the size estimate is determined based on a prior size estimate from a prior frame, and the 3D position is determined based on the size estimate of the representation of the object and a portion of the image data.

Translation: The system keeps track of objects in 3D by relying on historical size data whenever the depth sensors stop working for a specific frame.

How the system fills gaps when depth data drops out

The patent describes a frame-by-frame pipeline for estimating an object's 3D position and physical size using a mix of camera images and depth data.

When a frame includes both image data and depth data (from something like a LiDAR or time-of-flight sensor), the system does the full calculation: it uses the device's known orientation and position in space (pose data, essentially where the phone is pointing and how it's tilted) together with the depth reading to pin down exactly where the object is, and it uses the depth plus the image to calculate how large the object appears to be in the real world.

When a frame has image data but no depth data, the system does not give up. Instead:

  • It pulls the size estimate from the most recent frame that did have depth data (the "prior size estimate").
  • It uses that known size as a reference anchor, then reads the object's apparent size in the current camera image to back-calculate how far away the object must be.
  • From that inferred distance, it reconstructs the object's 3D position without needing fresh depth sensor input.

The result is a continuous stream of position and size estimates even when the depth sensor misses frames, which happens regularly on consumer devices during fast movement or in low-contrast scenes.

What this means for AR on iPhones and Vision Pro

Depth sensors on iPhones and the Vision Pro headset are useful but imperfect. They miss frames, they struggle at certain distances, and they drain power. Any AR or measurement feature that depends on depth alone will stutter when the sensor stumbles. Apple's approach treats depth data as a helpful input when available, not a hard requirement on every single frame, which makes the whole system more resilient in real conditions.

For consumers, this points toward more reliable versions of features like the Measure app or Vision Pro's spatial awareness. For developers building AR experiences, a steadier object-tracking foundation means fewer edge cases to code around. Apple's run of spatial-computing filings shows the company treating 3D object understanding as infrastructure, not a feature.

Apple's 75th filing we've tracked in our AR glasses work since May adds to a run that includes one on shifting 3D photo depth and one on varied meditation audio.

Source: US 2026/0301215 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Apple patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 7 sheets

7 drawing sheets from US 2026/0301215 A1 · click any drawing to enlarge

People interacting with augmented reality., drawing sheet 1 from US 2026/0301215 A1
Sheet 1 People interacting with augmented reality.
Augmented reality view of a soccer game., drawing sheet 2 from US 2026/0301215 A1
Sheet 2 Augmented reality view of a soccer game.
System for tracking objects in 3D., drawing sheet 3 from US 2026/0301215 A1
Sheet 3 System for tracking objects in 3D.
Steps for determining object position and size., drawing sheet 4 from US 2026/0301215 A1
Sheet 4 Steps for determining object position and size.
Steps for determining object location and velocity., drawing sheet 5 from US 2026/0301215 A1
Sheet 5 Steps for determining object location and velocity.
Components of a processing device., drawing sheet 6 from US 2026/0301215 A1
Sheet 6 Components of a processing device.
Headset with cameras and display., drawing sheet 7 from US 2026/0301215 A1
Sheet 7 Headset with cameras and display.
Patent filing page

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.