Google Patents a Way to Make Video AI Focus on the Right Objects
Most AI that watches video wastes processing power on empty walls and parked cars. Google has filed a patent for a system that teaches video-recognition AI to pay more attention to the things that actually matter, like the people and objects in the scene.
How Google's video AI learns to ignore the background
Imagine you're watching a security camera feed of a busy parking lot. Most of the frame is empty asphalt, but what you actually care about is the person walking toward a car. A good human guard focuses on that person automatically. Current AI systems, however, treat every pixel of the frame with roughly equal effort, which is slow and often leads to missed actions.
Google's patent describes a method for giving video AI a similar sense of priority. The system first uses an object detector (a separate tool that draws boxes around recognizable things like people, hands, or cars) to mark what's important in each frame. Regions inside those boxes get labeled as "foreground" and receive full attention from the AI. Everything else, the background, gets heavily compressed before the AI even looks at it.
The result is a video-understanding system that spends its computing budget where it counts. That could make action-recognition AI faster, cheaper to run, and more accurate, whether it's powering video search on YouTube or flagging unsafe activity in a warehouse.
… identifying, by the one or more processors based on the object detections, a set of foreground tokens and a set of background tokens according to object locations in a frame of the plurality of video frames, wherein each foreground token and each background token is a nonoverlapping space-time token; …
Translation: The system sorts the video data into distinct foreground and background pieces based on where objects appear.
How foreground tokens and downsampled background tokens work together
The patent describes a pipeline built around a transformer (a type of AI model that processes data by figuring out which parts of the input are most related to each other). The transformer looks at video as a grid of small patches, called tokens, spread across space and time.
Here is how the process works step by step:
- Object detection first: Before the transformer does anything, a separate detector identifies where objects are located in the video frames, producing bounding boxes.
- Token sorting: Based on those boxes, every space-time patch token in the video is labeled either a foreground token (inside a detected object's region) or a background token (everything else). Crucially, no two tokens overlap.
- Background downsampling: The background tokens are compressed into a much smaller set, discarding redundant information about walls, floors, and sky.
- Object-aware attention: The full foreground set and the reduced background set are both fed into an object-aware attention module. Attention (think of it as the AI deciding where to look) then updates the representation of each patch based on this curated input.
The updated patch features are then handed off to whatever task the system needs to complete, whether that is classifying an action ("person picking up a box"), detecting an event, or something else. The key insight is that injecting external object locations into the transformer's attention process makes the AI's "gaze" more purposeful from the start.
The system downsamples the set of background tokens to obtain a reduced set of background tokens ( 2106 ). The system applies the set of foreground tokens and the reduced set of background tokens to an object-aware attention module …
Translation: It shrinks the background data to help the AI focus its attention on the important objects.
What this means for video search and content moderation
For everyday users, the most visible payoff could come from video search. When you search YouTube for "basketball three-pointer" or "someone tripping," the AI behind those results has to watch millions of clips and figure out what is happening. A system that focuses on the players rather than the stands could return better results faster, and potentially at lower energy cost per query.
Google's interest in video understanding shows up across many of its products, from YouTube recommendations to Google Photos' automatic highlight reels. If this approach works as described, it could also benefit real-time systems like moderation tools or accessibility features that auto-caption video, where both speed and accuracy matter.
Google's 35th filing in the AI vision work we've tracked since May adds to a run that includes learning and copying camera angles and face recognition for home cameras.
Claim 1 is fairly broad. It covers any computer-implemented method that sorts video patches into foreground and background using object detections, compresses the background, then runs both through an attention module to update patch features. The claim does not restrict which object detector you use, what kind of transformer, or what the downstream video task is. That breadth means, if granted, it could apply to a wide range of video-AI pipelines that follow this general shape.
In practice, the interesting question is how much of that breadth survives examination. The core idea of prioritizing detected objects over background in a transformer is intuitive enough that prior art from 2022 to 2024 in the video-understanding literature could create friction. The patent's specific combination of nonoverlapping space-time tokens plus downsampled background plus an object-aware attention module is where any real novelty lives, and claim 1 only loosely describes those mechanics.
For readers tracking AI video tools, the filing is a reasonable signal of where Google is putting engineering effort. It does not announce a product, but it does describe a real problem (transformers treating every pixel equally) and a concrete architectural response.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
29 drawing sheets from US 2026/0279047 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in