Nvidia Patents an AI That Reads Text Instructions to Move Robot Arms in 3D
What if a robot arm could read "pick up the red cup" and figure out exactly where in 3D space to move, without a human programming every coordinate? That is the core idea in this Nvidia patent.
How Nvidia teaches robots to follow written orders
A robot arm sits in a warehouse, surrounded by boxes. A worker types "move the package to the left shelf." Right now, making that work reliably is genuinely hard, and most robots need highly structured environments and hand-coded instructions to pull it off.
Nvidia's approach lets a robot read plain text instructions the same way a language model reads a sentence, then look at camera images of the scene and predict where it needs to move. The system produces a kind of heat map showing the most likely target positions, then converts those 2D guesses from multiple camera angles into a single precise 3D location.
For you, this matters because it points toward robots that workers could direct in plain language rather than through specialized programming, making flexible automation more practical for the kinds of messy, changeable environments that real workplaces involve.
… generating, by a large language model (LLM) and based on the plurality of input tokens, a plurality of action tokens …
Translation: The system uses an AI language model to translate visual and text inputs into movement commands.
How the AI converts camera views into 3D robot positions
The system takes two inputs: one or more camera images of a scene, each captured from a specific pre-set angle, and a plain text instruction describing what the robot should do.
Those inputs get converted into tokens (chunks of data that a language model can process, similar to how words become tokens in a chatbot). A large language model (LLM) then processes those tokens and outputs a set of action tokens, which encode where the robot or its end-effector (the gripper or tool at the end of the arm) should be in its next state.
The action tokens are decoded into output images, one per camera viewpoint. Each output image is essentially a probability heatmap (a picture where bright spots indicate "the robot's hand is probably going here") rather than a single hard-coded coordinate. Using heatmaps instead of exact numbers is a known technique for handling uncertainty gracefully.
Finally, the system triangulates across those per-view heatmaps to compute an actual 3D position. Having multiple camera views means the system can resolve the ambiguity that any single camera angle creates, because a spot that looks close from one angle can look far from another.
… one or more output images indicate probability distributions of a location of the robot or an end-effector of the robot in a future state of the robot …
Translation: The AI outputs pictures showing the most likely future positions for the robot arm.
What this means for warehouse and factory robots
Factories and warehouses are full of situations where a robot needs to handle something it has never seen before, or respond to a spoken or typed request from a human. Right now, that usually requires expensive custom programming. A system that turns a text prompt into a precise 3D movement target could lower that barrier significantly, letting workers redirect robots on the fly without calling in a specialist.
Nvidia's steady investment in robot-training AI has been building toward exactly this kind of flexible, language-driven control. If this approach works at scale, the practical payoff is a robot that a line worker can instruct in ordinary language and that adapts when the layout or task changes, rather than one that breaks the moment something is out of place.
That makes this Nvidia's 30th filing we've tracked since May in our robot grasping and movement watchlist, adding to work like tracking object movement in video and one model for any robot.
The practical promise for someone working alongside a robot is the ability to give instructions in plain language rather than through careful reprogramming. A worker could describe a new task and the robot would interpret both the words and what it sees in the room, then figure out where to move next.
The heatmap approach matters because it lets the robot express uncertainty about where exactly to reach or grasp, rather than committing to a precise point that turns out to be slightly wrong. In everyday terms, the robot can aim for a region and refine from there, which is closer to how people actually handle unfamiliar situations.
The gap between a promising design and a robot that reliably works in a real warehouse or kitchen is substantial, and this filing describes the former. The benefit to a user is real in concept, but they would only feel it once the system holds up under the messy, unpredictable conditions of actual work.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
10 drawing sheets from US 2026/0268120 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →