Google Files Patent for Captions That Know Exactly When Each Word Was Said
Speech-to-text can get nearly every word right and still show your captions at the wrong moment. A new Google patent application tries to fix that by making the AI write down the time of each word as it transcribes.
What Google's timestamped speech training does
Ever watched a video where the captions show up a beat late, or race ahead of the person talking? It's more than annoying. When words appear off-beat, you lose the thread, and in a meeting transcript you can't easily click a line and jump to the moment it was said.
Google's patent application goes after the root of the problem. Many speech-to-text systems are good at working out what was said but lose track of when. Google's fix is to teach the system to write the time down as part of the transcript itself, so each word comes out with a timestamp attached, like 'Hello' ending at 200 milliseconds.
During training, the system studies recordings paired with transcripts that already include those times, and it gets corrected whenever it misses the words or the timing. Once trained, one model hands back both the text and the timing, with no separate step to line them up afterward.
obtaining training data comprising a plurality of training utterances each paired with a corresponding target output sequence, each corresponding target output sequence comprising a sequence of words and associated timestamps …
Translation: The system collects voice recordings that are already matched with exact word timings.
How Google writes word times into transcripts
The patent covers a way to train a speech recognition model. Each training recording is paired with an answer key holding the words and a timestamp for each one. The model has an audio encoder, which turns sound into numbers a computer can work with, and a decoder, which writes out the text. The filing says the decoder can be a large language model.
After each recording, a loss (a score of how far the output is from the answer key) guides how the model is adjusted. Wrong words and wrong times both count against it. Timestamps use their own symbols, kept apart from word pieces, so a spoken '200' in 'read 200 pages' can't be mistaken for 200 time units. Each one attaches right after its word with no space, which keeps the output short.
Times are rounded to a fixed unit, such as 10 milliseconds, and written in a base bigger than ten (the filing mentions base-100), so a value like 500 ms fits in a single symbol. The answer key can also mark pauses, so silence between 'Hello' and 'world!' isn't credited to the next word.
If recordings lack timing, an alignment model (a tool that matches audio to a transcript) can add it first. A plain prompt can then ask the model to include timestamps.
Each corresponding target output sequence includes a sequence of words and associated timestamps. Each timestamp indicates a timing associated with one of the words in the sequence of words of the target output sequence.
Translation: Every sentence comes with specific markers showing the exact moment each word was spoken.
Why this matters for captions and meeting notes
The practical payoff is captions and transcripts that line up with the audio. The filing points to video captions, meeting transcripts you can click to jump to the moment someone spoke, and searchable audio and video archives. It says earlier tricks for pulling timing out of this kind of model could be off by hundreds of milliseconds, enough to make captions feel late.
The filing also frames this as a way to get timing from a model built for fluent, accurate text, without a separate step afterward. If you watch captioned video or read auto-generated meeting notes, this is a behind-the-scenes fix you would notice only when it's missing. The filing names no product, so treat it as a technical building block.
Google's 62nd entry in the voice and speech AI work we've tracked since May connects to earlier ideas like learning from its own guesses and fast typing that self-corrects.
Claim 1 is short and wide. It asks for training recordings paired with transcripts where each word carries a timestamp, a piece that turns sound into numbers (the audio encoder), a piece that writes out the result (the decoder), and a score for how far off the output is. That's the ordinary recipe for training a speech model, with one change: the answer key includes times.
The specific tricks, such as pause markers, separate time symbols, a fixed time unit, a prompt asking for timestamps and an alignment tool, show up only in the later claims. Claim 1 doesn't demand any of them. If granted as written, that breadth would cover on its face any speech model trained with word-timed answer keys, whatever shape the time symbols take.
Whether a claim this plain survives is for the patent office to decide. For you, the upshot is simple: the timing lives inside the transcript, and that's a practical fix for late captions.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
5 drawing sheets from US 2026/0351246 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in