Nvidia Patents a Speaker-Memory System That Tracks Who's Talking in Real Time
Figuring out who said what in a multi-person conversation is one of the trickiest problems in audio AI. Nvidia's new patent describes a system that does it live, as audio streams in, by keeping a running memory of every speaker it has heard so far.

- Publication number
- US 2026/0245574 A1(US20260245574A1)
- Applicant
- NVIDIA Corporation
- Filed · Published
- Feb 18, 2026 · Aug 20, 2026
- Inventors
- Ivan Medennikov, Taejin Park, Weiqing Wang, He Huang, Kunal Dhawan, Jinhan Wang, Myungjong Kim, Jagadeesh Balam, Boris Ginsburg
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 704/232
- Parent application
- Claims priority from a provisional application 63760485 (filed Feb 19, 2025)
How Nvidia's speaker-tracking cache actually works
A doctor dictates notes while two colleagues chime in. A podcast host and three guests talk over each other. A court reporter captures six people arguing at once. In all these cases, the audio software needs to know not just what was said, but who said it.
Nvidia's patent describes a system that tackles this in real time rather than waiting until a recording is finished. It processes speech in small chunks as they arrive, while maintaining a "speaker cache", a running memory log of every voice it has already heard. That cache is organized by the order speakers first showed up, so the system always knows who was speaking without having to re-analyze the whole recording from scratch.
The result is a speaker-to-time map: a labeled timeline that says "Speaker 1 talked here, Speaker 2 talked there." The cache gets updated with each new chunk of audio, so the system stays current as the conversation keeps going.
processing, using a trained speaker diarization model, an input comprising (i) one or more digital units (DUs) of a segment of speech and (ii) a speaker cache of one or more previously processed DUs of the speech to obtain a plurality of likelihood values …
Translation: The system uses a memory bank of past audio clips to help identify who is currently speaking.
Inside the digital-unit pipeline and cache update loop
The system is built around a trained speaker diarization model, an AI that has learned to distinguish between voices and assign them identities (Speaker 1, Speaker 2, and so on). What makes this patent distinct is the mechanism for doing that job as audio arrives, rather than in post-processing.
Audio is broken into digital units (DUs), small, bite-sized segments of the incoming speech stream. For each new segment, the model receives two inputs:
- The current DU (or a short batch of DUs) representing the latest slice of audio
- The speaker cache, a stored representation of previously processed DUs, arranged so that each position in the cache corresponds to the order in which a new speaker first appeared
The model outputs a set of likelihood values: confidence scores for which speaker (or speakers) is talking at any given moment. Those scores are then used to update the cache, refreshing its memory of each speaker's voice characteristics based on the newest audio.
Finally, the system assembles all of these likelihood scores into a speaker-to-time mapping: a labeled timeline showing which speaker was active at each point in the recording. Because the cache is indexed by order of appearance rather than by a fixed speaker ID, the system handles new speakers joining mid-conversation without having to restart or reprocess old audio.
The techniques further include updating the speaker cache using at least the one or more DUs of the segment of the speech and the plurality of likelihood values. The techniques further include generating, using the plurality of likelihood values, a speaker-to-time mapping for the speech.
Translation: The software constantly updates its memory to create a timeline showing exactly when each person speaks.
What this means for real-time transcription and voice AI
Real-time speaker tracking has been a long-standing weak point for transcription tools, meeting assistants, and voice analytics platforms. Most accurate diarization systems today work only after a recording ends, which makes them useless for live captioning, real-time translation, or any application where you need to know who is speaking right now. A cache-based streaming approach, if it works reliably, would bring that accuracy to live scenarios.
For Nvidia, whose hardware already powers a large portion of the world's speech AI workloads, a patented diarization method could slot into its Riva speech AI platform or similar enterprise offerings. The plain-English patent summaries Patentlyze tracks across voice and audio AI show this filing as part of a broader push to bring real-time speaker intelligence to the same tier of accuracy that offline processing has enjoyed for years.
Nvidia's 403rd filing among the patents we've tracked in our Nvidia coverage since May joins earlier work on turning 3D scans into digital models and remote control for AI cameras.
Source: US 2026/0245574 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 17 sheets
17 drawing sheets from US 2026/0245574 A1 · click any drawing to enlarge
