Nvidia Files Patent to Stop AI From Forgetting Text While Learning Speech
Teach an AI to understand speech the usual way and it can get worse at the text work it already knew. Nvidia's patent application tries to avoid that by mixing both kinds of lessons from the start.
What Nvidia's mixed speech-and-text training does
Today, teaching a text-based AI model to understand speech usually happens in stages: train it on text first, then add speech skills afterward. The catch is that the model often gets worse at the text work it already knew, a problem researchers call catastrophic forgetting. The common patch is to freeze the original model and attach a small add-on, which makes the whole setup clunkier to update.
Nvidia's filing takes a simpler route. Instead of finishing one skill before starting the next, it shuffles the lessons together. The model sees a batch of text examples, adjusts itself, then sees a batch of spoken examples, adjusts again, and keeps alternating. Think of a student who practices vocabulary and listening every day, rather than cramming one for a month and the other the next.
The filing also describes a way to pack each batch so the computer chips doing the work don't waste effort or run out of memory.
selecting one or more first units of training data of a first modality to generate a first training batch; selecting one or more second units of training data of a second modality to generate a second training batch; …
Translation: The system groups data from two different media types into separate batches for training.
How the test runs size each training batch
Claim 1 covers a simple recipe. Pick a batch of training data from one modality (a type of input, such as text or speech), then a batch from a second modality. Run each batch through a language model (LM) to work out how its parameters (the internal numbers that store what it has learned) should change. Then update those parameters using both changes.
The filing describes two ways to apply the changes:
- Apply each batch's adjustment right away, before the next batch is processed.
- Store each adjustment, add them together after both batches are done, and apply the combined change once.
Which batch comes next can be chosen at random, weighted by how much data of each type is left, so both run out at about the same time.
The second half is about efficiency. Training examples are sorted into bins by input length and expected output length, so short items don't get padded with zeros (wasted math). To find how many items fit in a batch, the system runs dummy batches of random data and measures GPU memory use. If memory is underused it grows the batch, and if it is overused it shrinks it, using finer steps until it lands near a target such as 95%. Independent claim 12 covers this batch-sizing step on its own, for any AI model.
Disclosed are apparatuses, systems, and techniques that implement interleaved multi-modal training for training of language models.
Translation: Nvidia invented a method to train language models using mixed types of data without losing prior knowledge.
Why forgetting is costly for voice assistants
If you have used live captions, a voice assistant in your car, or a meeting tool that translates on the fly, you have used software that must handle both speech and writing. A model that picks up speech by losing some text skill is a worse product, and the filing's mixed training targets that tradeoff. The description also says single-stage training makes it easier to retrain for a new field, such as one full of rare or unusual words.
The batch-sizing part is about cost. Training these models ties up expensive GPUs (graphics chips), and the filing says padding short items with zeros makes the chips spend time on empty math. Sorting by length and filling chip memory close to the limit is housekeeping, but it is housekeeping on a very large bill.
This is the 25th voice and speech AI filing from Nvidia we've tracked since May, a group that includes one on reading texts in the sender's voice and one on fixing NVIDIA transcription errors.
Claim 1 is short and wide. It asks for a batch from one type of data, a batch from a second type, a language model that learns from each, and an update that uses both. It never names speech, text, random picking or any particular chip; those details arrive in later claims.
Mixing two kinds of examples in one training run sounds like a natural step, so with wording this open the narrower claims are likely to matter more. Those cover choosing batches by how much data is left, adding up the adjustments, and sorting examples by length.
The most concrete idea is claim 12, which stands alone: run dummy batches, watch memory use, and resize until the chip is nearly full. It applies to any AI model, and it is the piece most likely to save real computing time.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
19 drawing sheets from US 2026/0340379 A1 · click any drawing to enlarge
Show the other 11 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in