Meta Patents AR Glasses That Watch What You're Doing and Coach You in Real Time
Meta is patenting a system where your AR glasses watch what you're doing — using their outward-facing camera — and layer real-time tips or corrections right into your field of view.
What Meta's AR action-coaching system actually does
Imagine you're following a recipe and your AR glasses notice you're chopping vegetables the wrong way. Instead of you stopping to check a tutorial on your phone, the glasses just… show you the correction, floating right over what you're looking at. That's the core idea here.
Meta's patent describes AR glasses that can detect when you — the wearer — appear in the device's outward camera view (think a mirror, a storefront window, or a second camera angle), figure out what action you're performing, and then supplement your view with relevant guidance or feedback.
The loop is automatic: spot the user, identify the activity, respond with on-screen help. No tapping, no voice command — just contextual coaching that shows up when the glasses decide you need it.
How the AR device detects you and overlays guidance
The patent describes a three-step detection-and-response pipeline running on an AR headset — almost certainly aimed at something in the Ray-Ban Meta smart glasses family or a future heads-up display device.
- Step 1 — User detection: The outward-facing camera watches the scene. When it spots the wearer themselves (via a mirror, reflective surface, or second-person view), it flags that the user is now visible in the frame.
- Step 2 — Action recognition: Once the user is detected, the system classifies what action they appear to be performing — think exercise form, a cooking step, a hand gesture, or a work task.
- Step 3 — View supplementation: Based on the identified action, the device overlays additional content into the AR view area — guidance text, corrective arrows, instructional overlays, or other contextual feedback.
The patent leans on the device's onboard sensors doing this detection loop locally, which keeps latency low and avoids needing a constant cloud round-trip. The abstract is light on specifics about the AI model doing the action recognition, but the framing implies computer vision running on-device — consistent with Meta's investment in edge inference for its wearables stack.
What this means for Meta's Ray-Ban glasses roadmap
For Meta, this patent is a concrete step toward making AR glasses useful rather than just ambient. The biggest knock on first-gen smart glasses is that they're passive — they record and play audio, but they don't respond to what you're actually doing. A system that watches your actions and proactively coaches you flips that dynamic entirely.
If this ships in something like a future Ray-Ban Meta or a full AR headset, it could unlock fitness coaching, workplace training, cooking assistance, and accessibility use cases — all hands-free. The key unlock is the self-detection piece: using a reflective surface as a trigger is an elegant way to know the wearer is paying attention to themselves, which is exactly when real-time feedback is most useful.
This is one of Meta's more practically grounded AR patents — it solves a real problem (when do glasses know to help you?) with a clever trigger (detect yourself in a mirror). The reflective-surface detection approach is the interesting bit here; it sidesteps the creepy always-watching framing by anchoring guidance to a moment of intentional self-review. Whether the action-recognition accuracy is good enough to make this delightful rather than annoying is entirely an execution question.
Get one Big Tech patent every Sunday
Plain English, intelligent commentary, no hype. Free.
Editorial commentary on a publicly published patent application. Not legal advice.