Nvidia Patents a Speaker-Memory System That Tracks Who's Talking in Real Time
Figuring out who said what in a multi-person conversation is one of the trickiest problems in audio AI. Nvidia's new patent describes a system that does it live, as audio streams in, by keeping a running memory of every speaker it has heard so far.
How Nvidia's speaker-tracking cache actually works
A doctor dictates notes while two colleagues chime in. A podcast host and three guests talk over each other. A court reporter captures six people arguing at once. In all these cases, the audio software needs to know not just what was said, but who said it.
Nvidia's patent describes a system that tackles this in real time rather than waiting until a recording is finished. It processes speech in small chunks as they arrive, while maintaining a "speaker cache", a running memory log of every voice it has already heard. That cache is organized by the order speakers first showed up, so the system always knows who was speaking without having to re-analyze the whole recording from scratch.
The result is a speaker-to-time map: a labeled timeline that says "Speaker 1 talked here, Speaker 2 talked there." The cache gets updated with each new chunk of audio, so the system stays current as the conversation keeps going.
processing, using a trained speaker diarization model, an input comprising (i) one or more digital units (DUs) of a segment of speech and (ii) a speaker cache of one or more previously processed DUs of the speech to obtain a plurality of likelihood values …
Translation: The system uses a memory bank of past audio clips to help identify who is currently speaking.
Inside the digital-unit pipeline and cache update loop
The system is built around a trained speaker diarization model, an AI that has learned to distinguish between voices and assign them identities (Speaker 1, Speaker 2, and so on). What makes this patent distinct is the mechanism for doing that job as audio arrives, rather than in post-processing.
Audio is broken into digital units (DUs), small, bite-sized segments of the incoming speech stream. For each new segment, the model receives two inputs:
- The current DU (or a short batch of DUs) representing the latest slice of audio
- The speaker cache, a stored representation of previously processed DUs, arranged so that each position in the cache corresponds to the order in which a new speaker first appeared
The model outputs a set of likelihood values: confidence scores for which speaker (or speakers) is talking at any given moment. Those scores are then used to update the cache, refreshing its memory of each speaker's voice characteristics based on the newest audio.
Finally, the system assembles all of these likelihood scores into a speaker-to-time mapping: a labeled timeline showing which speaker was active at each point in the recording. Because the cache is indexed by order of appearance rather than by a fixed speaker ID, the system handles new speakers joining mid-conversation without having to restart or reprocess old audio.
The techniques further include updating the speaker cache using at least the one or more DUs of the segment of the speech and the plurality of likelihood values. The techniques further include generating, using the plurality of likelihood values, a speaker-to-time mapping for the speech.
Translation: The software constantly updates its memory to create a timeline showing exactly when each person speaks.
What this means for real-time transcription and voice AI
Real-time speaker tracking has been a long-standing weak point for transcription tools, meeting assistants, and voice analytics platforms. Most accurate diarization systems today work only after a recording ends, which makes them useless for live captioning, real-time translation, or any application where you need to know who is speaking right now. A cache-based streaming approach, if it works reliably, would bring that accuracy to live scenarios.
For Nvidia, whose hardware already powers a large portion of the world's speech AI workloads, a patented diarization method could slot into its Riva speech AI platform or similar enterprise offerings. The plain-English patent summaries Patentlyze tracks across voice and audio AI show this filing as part of a broader push to bring real-time speaker intelligence to the same tier of accuracy that offline processing has enjoyed for years.
The core claim here is broad. It does not tie the invention to one specific design or audio format. Instead, it covers the general idea of feeding live sound into a system that tracks and labels who is speaking, while keeping a running list of all the speakers heard so far. Any competitor building a similar "who said what" tool for live audio would have to work hard to avoid stepping on this patent.
How much this matters depends on whether the patent office forces the company to narrow that claim during review. But as written, it grabs a wide swath of territory in the growing business of automatically sorting out speakers during live conversations.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
17 drawing sheets from US 2026/0245574 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →