Google Patent: Robots Obeying Open-Ended Object Commands Without Hardcoded Vocabulary Lists
Most robots today need to be trained on every object they'll ever touch. Google's new patent describes a system that lets a robot read a plain-English instruction, figure out where the relevant objects are in a camera image, and then decide what to do, without being hard-coded for any particular thing.
How Google wants robots to act on plain-English commands
Today's robots are brittle: tell one to 'pick up the red mug' and it can only do that if it was specifically trained on red mugs. Swap in a blue thermos and it's lost. Google wants to change that by giving robots a more flexible way to understand instructions.
The idea is to split the robot's brain into two cooperating parts. The first part looks at whatever the camera sees and finds the objects mentioned in the instruction, locating them in the image. The second part takes that location information plus the original instruction and decides what physical action the robot should take next.
Because the object-finding part is driven by natural language rather than a fixed list, the robot can theoretically handle any object you describe in words, even ones it has never physically encountered before. That's the 'open-vocabulary' promise: the robot's vocabulary is as wide as human language.
… processing the object localization input using an object localization neural network to generate an object localization output that identifies respective locations of the one or more objects in the observation image …
Translation: An AI network analyzes the scene to pinpoint where targeted objects are located.
How the two neural networks split the job between them
The patent describes a pipeline with two neural networks working in sequence.
Step one: object localization. When the robot receives a text instruction like 'place the ceramic bowl near the sink,' the system extracts a natural-language phrase describing the objects (here, 'ceramic bowl' and 'sink') and feeds that phrase alongside a camera image into an object localization neural network (think of it as a visual search engine that highlights where named things appear in a photo). The output is a map of bounding boxes or coordinates pinpointing each object.
Step two: action policy. A second network, called the policy neural network, receives three things: the original camera image, the full text instruction, and the object-location map from step one. It processes all of this together and outputs a specific physical action, such as 'move arm 4 cm forward and close gripper.' The robot then executes that action.
The key distinction from older robotic systems is that the object-finding step uses open-vocabulary recognition, meaning it is not restricted to a closed list of objects seen during training. Because it interprets a natural-language description, it can generalize to objects described in words even if they look different from anything in the training data. The policy network then uses the spatial information to ground its decision in the actual scene in front of the robot.
… processing the policy input using a policy neural network to generate a policy output that defines an action to be performed by the agent in response to the observation image …
Translation: A second AI model decides what physical move the robot should make next.
What this means for real-world robot deployment
For anyone building robots that need to work in unpredictable environments, like warehouses, kitchens, or hospital rooms, this approach addresses a real bottleneck. Right now, deploying a robot in a new setting usually requires expensive re-training every time a new object category shows up. A system that can locate and interact with objects described only in plain language would dramatically reduce that overhead.
Google has been one of the most active companies in robot-learning research, and this patent fits a pattern of using large language and vision models as the 'brain' behind physical machines. The approach described here is software-first: it does not require new robot hardware, only a camera and enough onboard computing to run two neural networks, which makes the path to a shippable feature shorter than many robotics patents. Among the interesting tech patents in the robotics and AI-control space, Google's open-vocabulary framing signals a deliberate push toward robots that operators can instruct in everyday language rather than pre-programmed commands.
That makes this Google's second filing we've tracked on teaching robots to grasp and move since August, following their self-steering robot arm application.
The patent describes two software systems working together: one that spots objects in a camera image, and one that decides what a robot arm should do next. No new physical hardware is required, which matters because any robot already equipped with a camera and a capable computer could, in principle, run this.
The shortest path to a product runs through reliability. The document does not address what happens when the camera view is blocked, how the system recovers from a wrong guess, or whether it responds fast enough for a busy warehouse. Those are the gaps a manager would notice before signing a purchase order.
The building blocks described here already exist in research settings, and the filing lays out a clear method for connecting them to physical robot movement. The remaining work is integration and testing, which is finishable work rather than a new scientific problem.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0249453 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →