Get the newsletter
Nvidia · AI / ML · Filed Feb 18, 2026 · USPTO published Aug 20, 2026 · Aug 20 release

Nvidia Patents a Speaker-Memory System That Tracks Who's Talking in Real Time

Figuring out who said what in a multi-person conversation is one of the trickiest problems in audio AI. Nvidia's new patent describes a system that does it live, as audio streams in, by keeping a running memory of every speaker it has heard so far.

Sequential timeline showing audio segments updating a speaker-mapped cache in real time. Drawing from patent filing US 2026/0245574 A1.
Sequential timeline showing audio segments updating a speaker-mapped cache in real time.All 17 drawings ↓
Publication number
US 2026/0245574 A1(US20260245574A1)
Applicant
NVIDIA Corporation
Filed · Published
Feb 18, 2026 · Aug 20, 2026
Inventors
Ivan Medennikov, Taejin Park, Weiqing Wang, He Huang, Kunal Dhawan, Jinhan Wang, Myungjong Kim, Jagadeesh Balam, Boris Ginsburg
Status when we published
Waiting for an examiner
Document
20 claims
US classification
704/232
Parent application
Claims priority from a provisional application 63760485 (filed Feb 19, 2025)

How Nvidia's speaker-tracking cache actually works

A doctor dictates notes while two colleagues chime in. A podcast host and three guests talk over each other. A court reporter captures six people arguing at once. In all these cases, the audio software needs to know not just what was said, but who said it.

Nvidia's patent describes a system that tackles this in real time rather than waiting until a recording is finished. It processes speech in small chunks as they arrive, while maintaining a "speaker cache", a running memory log of every voice it has already heard. That cache is organized by the order speakers first showed up, so the system always knows who was speaking without having to re-analyze the whole recording from scratch.

The result is a speaker-to-time map: a labeled timeline that says "Speaker 1 talked here, Speaker 2 talked there." The cache gets updated with each new chunk of audio, so the system stays current as the conversation keeps going.

From the filing · CLAIM 1
processing, using a trained speaker diarization model, an input comprising (i) one or more digital units (DUs) of a segment of speech and (ii) a speaker cache of one or more previously processed DUs of the speech to obtain a plurality of likelihood values …

Translation: The system uses a memory bank of past audio clips to help identify who is currently speaking.

Inside the digital-unit pipeline and cache update loop

The system is built around a trained speaker diarization model, an AI that has learned to distinguish between voices and assign them identities (Speaker 1, Speaker 2, and so on). What makes this patent distinct is the mechanism for doing that job as audio arrives, rather than in post-processing.

Audio is broken into digital units (DUs), small, bite-sized segments of the incoming speech stream. For each new segment, the model receives two inputs:

  • The current DU (or a short batch of DUs) representing the latest slice of audio
  • The speaker cache, a stored representation of previously processed DUs, arranged so that each position in the cache corresponds to the order in which a new speaker first appeared

The model outputs a set of likelihood values: confidence scores for which speaker (or speakers) is talking at any given moment. Those scores are then used to update the cache, refreshing its memory of each speaker's voice characteristics based on the newest audio.

Finally, the system assembles all of these likelihood scores into a speaker-to-time mapping: a labeled timeline showing which speaker was active at each point in the recording. Because the cache is indexed by order of appearance rather than by a fixed speaker ID, the system handles new speakers joining mid-conversation without having to restart or reprocess old audio.

From the filing · THE ABSTRACT
The techniques further include updating the speaker cache using at least the one or more DUs of the segment of the speech and the plurality of likelihood values. The techniques further include generating, using the plurality of likelihood values, a speaker-to-time mapping for the speech.

Translation: The software constantly updates its memory to create a timeline showing exactly when each person speaks.

What this means for real-time transcription and voice AI

Real-time speaker tracking has been a long-standing weak point for transcription tools, meeting assistants, and voice analytics platforms. Most accurate diarization systems today work only after a recording ends, which makes them useless for live captioning, real-time translation, or any application where you need to know who is speaking right now. A cache-based streaming approach, if it works reliably, would bring that accuracy to live scenarios.

For Nvidia, whose hardware already powers a large portion of the world's speech AI workloads, a patented diarization method could slot into its Riva speech AI platform or similar enterprise offerings. The plain-English patent summaries Patentlyze tracks across voice and audio AI show this filing as part of a broader push to bring real-time speaker intelligence to the same tier of accuracy that offline processing has enjoyed for years.

Nvidia's 403rd filing among the patents we've tracked in our Nvidia coverage since May joins earlier work on turning 3D scans into digital models and remote control for AI cameras.

Source: US 2026/0245574 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 17 sheets

17 drawing sheets from US 2026/0245574 A1 · click any drawing to enlarge

System architecture for audio processing, drawing sheet 1 from US 2026/0245574 A1
Sheet 1 System architecture for audio processing
Computing device internal hardware, drawing sheet 2 from US 2026/0245574 A1
Sheet 2 Computing device internal hardware
Audio data processing pipeline, drawing sheet 3 from US 2026/0245574 A1
Sheet 3 Audio data processing pipeline
Diarization model training pipeline, drawing sheet 4 from US 2026/0245574 A1
Sheet 4 Diarization model training pipeline
Real-time speaker memory caching timeline, drawing sheet 5 from US 2026/0245574 A1
Sheet 5 Real-time speaker memory caching timeline
Speaker diarization training process, drawing sheet 6 from US 2026/0245574 A1
Sheet 6 Speaker diarization training process
Hardware structure and arithmetic logic, drawing sheet 7 from US 2026/0245574 A1
Sheet 7 Hardware structure and arithmetic logic
Neural network training framework, drawing sheet 8 from US 2026/0245574 A1
Sheet 8 Neural network training framework
Show the other 9 sheets
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 9
Sheet 9
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 10
Sheet 10
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 11
Sheet 11
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 12
Sheet 12
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 13
Sheet 13
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 14
Sheet 14
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 15
Sheet 15
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 16
Sheet 16
Nvidia Patents a Speaker-Memory System That Tracks Who’s Talking in Real Time patent drawing, sheet 17
Sheet 17
Patent filing page

More in AI assistant & agent

All 436 AI assistant & agent patents →

Watchlists this filing belongs to

All watchlists →