Get the newsletter
Nvidia · Chip · Filed Jun 9, 2026 · USPTO published Oct 1, 2026 · Oct 1 release

Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work

GPUs do their best work when thousands of tiny tasks run in parallel, but today's software has limited say over exactly how that traffic gets routed. Nvidia's new patent describes an API that hands that control back to the programmer.

A compute unit's grid of thread blocks is organized into block clusters, each containing multiple thread blocks. Drawing from patent filing US 2026/0300006 A1.
A compute unit's grid of thread blocks is organized into block clusters, each containing multiple thread blocks.All 73 drawings ↓
Publication number
US 2026/0300006 A1(US20260300006A1)
Applicant
NVIDIA Corporation
Filed · Published
Jun 9, 2026 · Oct 1, 2026
Inventors
Ze Long, Kyrylo Perelygin, Harold Carter Edwards, Gokul Ramaswamy Hirisave Chandra Shekhara, Jaydeep Marathe, Ronny Meir Krashinsky, Girish Bhaskarrao Bharambe
Status when we published
Waiting for an examiner
Document
20 claims
US classification
719/328
Parent application
Continuation of 17/955,070 (filed Sep 28, 2022)(17955070)

What Nvidia's thread-cluster scheduling API actually does

A modern GPU is basically a city with thousands of tiny workers. Right now, when a program sends work to the GPU, the GPU's own internal scheduler decides which worker handles which task, and the program mostly just waits. That arrangement is fast in general, but it can be inefficient for programs that know exactly how their work should be divided.

Nvidia's patent describes a programming interface (a formal way for software to talk to hardware) that lets a program specify the scheduling policy before the GPU starts working. Instead of the GPU guessing, your code can say: split these thread clusters (groups of parallel tasks) across the processing units this way, not however you feel like it.

The practical effect is that software developers writing GPU-intensive applications, think AI training or scientific simulations, get a new lever to pull. They can tune how work is distributed to match their specific problem, rather than hoping the GPU's defaults are a good fit.

From the filing · CLAIM 1
cause one or more block clusters, each block cluster comprising a plurality of blocks of threads, to be performed concurrently by a plurality of compute units of an accelerator, wherein performance of the blocks of threads is distributed between the plurality of compute units according to a scheduling policy indicated to the API.

Translation: Software can now tell the graphics chip how to break up and run tasks at the same time.

How the API routes thread blocks across compute units

The patent centers on a single idea: an API call (a standardized instruction a program sends to the hardware) that includes a scheduling policy as one of its parameters.

Here's the structure the patent describes:

  • Block clusters: the GPU's work is organized into clusters of "blocks," where each block is itself a bundle of parallel threads (individual instructions running simultaneously). Think of a block as a team and a cluster as a division of teams.
  • Compute units: the physical processing sections of the GPU that actually execute those threads. A modern GPU can have hundreds of them.
  • Scheduling policy: the rule the GPU follows when deciding which compute unit handles which cluster. This is what the new API exposes. Instead of that rule being hidden inside the GPU driver, the calling program declares it explicitly.

The patent is written around CUDA, Nvidia's programming platform for GPU computing (the standard toolkit most AI and scientific software uses). So this isn't a hypothetical future interface; it's designed to slot into an ecosystem that millions of developers already work in.

The core claim is that running these clusters concurrently (at the same time) across multiple compute units, with a programmer-specified distribution rule, produces outcomes that the GPU's own default scheduler might not achieve.

From the filing · THE ABSTRACT
Apparatuses, systems, and techniques to execute CUDA programs. In at least one embodiment, an application programming interface is performed to determine a scheduling policy of one or more blocks of one or more threads.

Translation: The patent describes new ways to run Nvidia programming code using custom scheduling rules.

What this means for GPU-heavy AI and HPC workloads

For most consumer software, this patent changes nothing you'd notice day-to-day. The GPU in your laptop will keep running games and video edits just fine without any of this. The audience here is the relatively small group of engineers writing the low-level code that underpins AI model training, data-center inference, and high-performance computing, exactly the markets where Nvidia makes most of its money.

For those developers, gaining explicit control over scheduling policy is meaningful. Poorly distributed work across compute units causes some units to sit idle while others are overloaded, wasting expensive hardware time. A scheduling API lets a program avoid that waste for specific, well-understood workloads. Whether Nvidia exposes this in a way that's easy to use, or buries it in a niche CUDA extension few people touch, will determine how much real-world impact it has.

Nvidia's 49th filing we've tracked since July in the AI chip wars watchlist follows earlier work on a compression math fix and formatting data in the connector, adding another piece to how it handles AI data.

Source: US 2026/0300006 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.

There are more where this came from

We read the new patents from Big Tech, carmakers and more and send you the ones worth knowing. Plain English, free, every week.

The drawings · 73 sheets

73 drawing sheets from US 2026/0300006 A1 · click any drawing to enlarge

Processor and graphics processor interaction., drawing sheet 1 from US 2026/0300006 A1
Sheet 1 Processor and graphics processor interaction.
Threads within a thread block., drawing sheet 2 from US 2026/0300006 A1
Sheet 2 Threads within a thread block.
Blocks within a compute unit grid., drawing sheet 3 from US 2026/0300006 A1
Sheet 3 Blocks within a compute unit grid.
Thread blocks in a compute unit., drawing sheet 4 from US 2026/0300006 A1
Sheet 4 Thread blocks in a compute unit.
Block clusters in a compute unit., drawing sheet 5 from US 2026/0300006 A1
Sheet 5 Block clusters in a compute unit.
Kernel launch process flow., drawing sheet 6 from US 2026/0300006 A1
Sheet 6 Kernel launch process flow.
Examples of block cluster dimensions., drawing sheet 7 from US 2026/0300006 A1
Sheet 7 Examples of block cluster dimensions.
Setting block cluster dimensions., drawing sheet 8 from US 2026/0300006 A1
Sheet 8 Setting block cluster dimensions.
Show the other 52 sheets
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 9
Sheet 9
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 10
Sheet 10
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 11
Sheet 11
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 12
Sheet 12
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 13
Sheet 13
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 14
Sheet 14
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 15
Sheet 15
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 16
Sheet 16
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 17
Sheet 17
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 18
Sheet 18
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 19
Sheet 19
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 20
Sheet 20
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 21
Sheet 21
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 22
Sheet 22
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 23
Sheet 23
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 24
Sheet 24
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 25
Sheet 25
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 26
Sheet 26
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 27
Sheet 27
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 28
Sheet 28
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 29
Sheet 29
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 30
Sheet 30
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 31
Sheet 31
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 32
Sheet 32
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 33
Sheet 33
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 34
Sheet 34
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 35
Sheet 35
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 36
Sheet 36
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 37
Sheet 37
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 38
Sheet 38
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 39
Sheet 39
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 40
Sheet 40
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 41
Sheet 41
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 42
Sheet 42
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 43
Sheet 43
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 44
Sheet 44
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 45
Sheet 45
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 46
Sheet 46
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 47
Sheet 47
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 48
Sheet 48
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 49
Sheet 49
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 50
Sheet 50
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 51
Sheet 51
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 52
Sheet 52
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 53
Sheet 53
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 54
Sheet 54
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 55
Sheet 55
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 56
Sheet 56
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 57
Sheet 57
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 58
Sheet 58
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 59
Sheet 59
Nvidia Patents an API That Lets Programs Control How a GPU Splits Its Work patent drawing, sheet 60
Sheet 60

Sheets 61 to 73 are in the full USPTO filing.

Patent filing page

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.