Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work
GPUs do their best work when thousands of tiny tasks run in parallel, but today's software has limited say over exactly how that traffic gets routed. Nvidia's new patent describes an API that hands that control back to the programmer.

- Publication number
- US 2026/0300006 A1(US20260300006A1)
- Applicant
- NVIDIA Corporation
- Filed · Published
- Jun 9, 2026 · Oct 1, 2026
- Inventors
- Ze Long, Kyrylo Perelygin, Harold Carter Edwards, Gokul Ramaswamy Hirisave Chandra Shekhara, Jaydeep Marathe, Ronny Meir Krashinsky, Girish Bhaskarrao Bharambe
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 719/328
- Parent application
- Continuation of 17/955,070 (filed Sep 28, 2022)(17955070)
What Nvidia's thread-cluster scheduling API actually does
A modern GPU is basically a city with thousands of tiny workers. Right now, when a program sends work to the GPU, the GPU's own internal scheduler decides which worker handles which task, and the program mostly just waits. That arrangement is fast in general, but it can be inefficient for programs that know exactly how their work should be divided.
Nvidia's patent describes a programming interface (a formal way for software to talk to hardware) that lets a program specify the scheduling policy before the GPU starts working. Instead of the GPU guessing, your code can say: split these thread clusters (groups of parallel tasks) across the processing units this way, not however you feel like it.
The practical effect is that software developers writing GPU-intensive applications, think AI training or scientific simulations, get a new lever to pull. They can tune how work is distributed to match their specific problem, rather than hoping the GPU's defaults are a good fit.
cause one or more block clusters, each block cluster comprising a plurality of blocks of threads, to be performed concurrently by a plurality of compute units of an accelerator, wherein performance of the blocks of threads is distributed between the plurality of compute units according to a scheduling policy indicated to the API.
Translation: Software can now tell the graphics chip how to break up and run tasks at the same time.
How the API routes thread blocks across compute units
The patent centers on a single idea: an API call (a standardized instruction a program sends to the hardware) that includes a scheduling policy as one of its parameters.
Here's the structure the patent describes:
- Block clusters: the GPU's work is organized into clusters of "blocks," where each block is itself a bundle of parallel threads (individual instructions running simultaneously). Think of a block as a team and a cluster as a division of teams.
- Compute units: the physical processing sections of the GPU that actually execute those threads. A modern GPU can have hundreds of them.
- Scheduling policy: the rule the GPU follows when deciding which compute unit handles which cluster. This is what the new API exposes. Instead of that rule being hidden inside the GPU driver, the calling program declares it explicitly.
The patent is written around CUDA, Nvidia's programming platform for GPU computing (the standard toolkit most AI and scientific software uses). So this isn't a hypothetical future interface; it's designed to slot into an ecosystem that millions of developers already work in.
The core claim is that running these clusters concurrently (at the same time) across multiple compute units, with a programmer-specified distribution rule, produces outcomes that the GPU's own default scheduler might not achieve.
Apparatuses, systems, and techniques to execute CUDA programs. In at least one embodiment, an application programming interface is performed to determine a scheduling policy of one or more blocks of one or more threads.
Translation: The patent describes new ways to run Nvidia programming code using custom scheduling rules.
What this means for GPU-heavy AI and HPC workloads
For most consumer software, this patent changes nothing you'd notice day-to-day. The GPU in your laptop will keep running games and video edits just fine without any of this. The audience here is the relatively small group of engineers writing the low-level code that underpins AI model training, data-center inference, and high-performance computing, exactly the markets where Nvidia makes most of its money.
For those developers, gaining explicit control over scheduling policy is meaningful. Poorly distributed work across compute units causes some units to sit idle while others are overloaded, wasting expensive hardware time. A scheduling API lets a program avoid that waste for specific, well-understood workloads. Whether Nvidia exposes this in a way that's easy to use, or buries it in a niche CUDA extension few people touch, will determine how much real-world impact it has.
Nvidia's 49th filing we've tracked since July in the AI chip wars watchlist follows earlier work on a compression math fix and formatting data in the connector, adding another piece to how it handles AI data.
Source: US 2026/0300006 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
We read the new patents from Big Tech, carmakers and more and send you the ones worth knowing. Plain English, free, every week.
The drawings · 73 sheets
73 drawing sheets from US 2026/0300006 A1 · click any drawing to enlarge
Show the other 52 sheets
Sheets 61 to 73 are in the full USPTO filing.

No comments yet