Nvidia Patent Automates Action and Relationship Labeling Within 3D Scene Graphs
Teaching an AI to understand a 3D world requires mountains of labeled data, and labeling that data by hand is slow and expensive. Nvidia's latest patent describes a system that builds a structured map of a 3D scene automatically, noting who is there, what they are doing, and where they stand in relation to everything else.
How Nvidia tags objects and actions in 3D spaces automatically
Imagine a security camera recording a parking lot. A person walks up to a car and opens the trunk. For a human watching, that sentence is obvious. For an AI, it requires knowing that the person is a subject, the car is an object, "opens the trunk" is the action connecting them, and the person is standing directly behind the vehicle. All of that context has to be labeled before the AI can learn from it.
Nvidia's patent describes a system that does this labeling automatically. It watches a 3D scene, identifies the people and objects in it, figures out what actions are happening, and notes the physical distances and positions involved. It then packages all of that into a structured diagram called a scene graph, where each thing in the scene is a dot and each relationship between things is a connecting line.
The result is a detailed, machine-readable description of a scene that an AI can train on without requiring a human to sit down and annotate every frame by hand. That kind of automated labeling is one of the bottlenecks holding back progress in robotics and self-driving vehicles.
How the 3D graph links subjects, objects, and spatial edges
The system processes a 3D scene and builds a graph data structure that captures three kinds of information about every subject it finds.
- Action edges: The system identifies what a subject (say, a person) is doing with respect to an object (say, a door) and records that as a connecting line, or "edge," between the two in the graph.
- Captions: Using the detected action alongside the visual appearance of the subject (drawn from bounding boxes, which are rectangular regions the system draws around detected objects in camera views), it generates a short text description of the subject. Think of this as an automatic label: "person carrying a box" rather than just "person."
- Spatial relationship edges: Using actual 3D coordinates, the system records where the subject is relative to the object: in front of, to the left of, above, and so on. This is a second type of connecting line in the graph.
All of this feeds into a 3D scene graph, a structured map where nodes represent things in the scene and edges represent how those things relate. The graph can describe a scene far more richly than a simple list of detected objects, because it encodes both actions and geometry in a single queryable structure.
What this means for self-driving cars and robotics training data
Labeled 3D data is the fuel that powers AI for robotics and autonomous vehicles. Right now, producing that data requires human annotators to watch footage and manually tag every object, every action, and every spatial relationship. That process is slow and costly, which is a real constraint on how fast companies can train capable AI systems.
For Nvidia, which sells the chips and software platforms that power both autonomous driving and robot training, making the data-creation pipeline faster and cheaper directly benefits its customers and, by extension, its own platform. A system that auto-generates rich scene descriptions could help developers build training datasets at a scale that would otherwise be impractical.
This is infrastructure work for AI training pipelines, not a consumer product. But it sits right at the center of one of the biggest practical problems in robotics and autonomous driving: getting enough good labeled data. Nvidia is essentially patenting an automated factory for one of the most tedious steps in that process, which is a strategically sensible place to plant a flag.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0228966 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Editorial commentary on a publicly published patent application. Not legal advice.