A New Nvidia Patent Merges AI Steps Together to Cut Wasted Data Shuffling
A lot of the time an AI chip spends isn't math at all, it's carrying results from one step to the next. Nvidia's patent describes software that finds the costliest hand-offs and glues those steps into one.
What Nvidia's layer-merging patent actually does
Ever watched a kitchen where one cook chops a carrot, walks it across the room to a second cook, who walks it to a third? Most of the time goes to walking, not cooking.
AI programs have the same problem. They are built from many small steps, and after each one the chip stores its results in memory, then fetches them again for the next step. That back-and-forth is slow and uses power.
Nvidia's patent describes a tool that prepares an AI program for the chip and decides which neighboring steps to combine into a single step. It measures how much data each hand-off would move, then merges the most expensive ones first. The combined step gives the same answers, but the data stays put instead of making the trip.
identify sets of nodes of a graph, wherein each node represents a processor-executable function; calculate cost values for combinations of nodes in the sets of nodes; select a set of nodes to combine based at least on the calculated cost values …
Translation: The system weighs how expensive different tasks are to run and decides which ones should be merged together.
How the compiler scores and merges AI steps
The patent describes a compiler (the tool that translates a developer's code into instructions a chip can run) that treats a neural network as a graph. Think of it as a map: each node (a dot) is one operation, such as a matrix multiplication, and each edge (a line) is the data flowing from one operation to the next.
The compiler first marks which connections can be fused, since it may only know how to generate code for some combinations. It then calculates a cost value for each one, based on how much data moves or has to be converted along that edge. From there the filing describes a loop:
- Rank the fusible connections by cost.
- Merge the nodes on the costliest connection into one bundle (called a partition).
- Recalculate the costs and repeat until nothing else can join, skipping any merge that would create a loop in the graph.
The code generated for a bundle gives the same result as running the steps separately, but it leaves data in place between them and avoids the transfers. The description calls the selection method a greedy heuristic, meaning it grabs the best-looking merge at each turn.
The claims cover this as a system, as a method, and as processor circuitry that does it in response to a software request (an API call). One claim even allows a merged set to include a node that doesn't share a direct connection with the others.
Apparatuses, systems, and techniques to perform a neural network. In at least one embodiment, a neural network is performed by at least combining nodes of a graph based, at least in part, on computing resources to perform operations corresponding to the nodes of the graph.
Translation: AI tasks are optimized by grouping steps according to the available computer hardware.
Why fewer memory trips matter for AI chips
If you use any app or service powered by AI, efficiency gains like this show up as faster answers and lower electricity bills for whoever runs the servers. The filing says its merged steps can cut run time, memory use and processor use, and it lists data centers and self-driving car computers among the places the approach could run.
The patent is a continuation of an application filed in January 2020, so it extends an idea with some history behind it. It also targets the software layer that sits between a developer's AI model and Nvidia's hardware, which is where the company decides how well its chips get used.
Nvidia's 52nd filing we've tracked since July in the AI chip wars builds on earlier applications like chips sharing math tasks and graphics and support chip data sharing.
Running AI costs a huge amount of chip time and electricity, and a surprising share of it goes to shuttling numbers around instead of calculating with them. Because every AI program has this overhead, even a small cut multiplies across an enormous number of runs. The problem is big enough to deserve attention.
The proposed fix is modest and fits that scale. Score each hand-off, merge the costliest first, and stop when nothing else can join. A rule that always grabs the biggest saving next won't always find the perfect arrangement, but it is fast and can run automatically on every program a developer builds.
That makes this a sensible piece of engineering for a widespread cost rather than a dramatic leap. You won't notice it directly, but it is the sort of change that makes AI services a little cheaper to run.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
58 drawing sheets from US 2026/0319775 A1 · click any drawing to enlarge
Show the other 50 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in