Google Files Patent to Stop AI Models From Watering Down Their Own Early Work
Google describes an AI design where every stage can look back at what several earlier stages produced, instead of only the one right before it. The filing says a 30-stage version beat a 42-stage standard one.
What Google's look-back AI layers actually do
Imagine writing a long report by passing it down a line of 30 editors, where each editor mostly works from the draft the previous person handed over. By the time it reaches the end, a sharp detail from the first editor may have been blurred by everyone in between.
Large AI models have a version of this problem. They are built from a tall stack of processing stages, and what an early stage figured out can get diluted as it moves up the stack. Google's filing lets each stage look back at the work of several earlier stages and choose how much of each to use.
The filing says this pays off. In its own tests, a 30-stage version scored better than a 42-stage standard version on a language benchmark, and training took up to 3 times fewer steps to reach comparable quality.
receive, by the generalized residual attention block, a block input that comprises a respective block output generated by each of two or more preceding attention blocks that are arranged preceding to the generalized residual attention block in the stack of the plurality of attention blocks; …
Translation: Each layer pulls in data from multiple earlier steps to keep older information from fading away.
How each layer picks which earlier work to reuse
Most modern AI models for text, images and audio are built from attention blocks stacked one after another. Attention is a way of deciding which parts of the input matter most for the step the model is working on. In a standard design, each block mostly works from the previous block's output, with earlier results simply added together.
Google's version, called a generalized residual attention block, receives the outputs of two or more earlier blocks at once. Three small helper pieces (one each for the query, key and value, the three views of the data that attention uses to pick its focus) blend those earlier outputs using weights learned during training. Each helper:
- Rescales the stacked outputs to a common range (layer normalization),
- Multiplies them by one shared set of weights and adds a separate set for each earlier block,
- Runs the result through a simple filter such as ReLU and uses it to weight the earlier outputs.
The filing also describes variations. A block could look at all earlier blocks, or only the first N and last K (for example 2 each, so at most 4 inputs whether the model has 10 blocks or 100). The main weights can start at zero so the model begins like a standard one, and the new blocks can be inserted into an existing trained model and fine-tuned.
The claims allow text, computer code, images, video or audio as inputs and outputs.
Why cheaper AI training matters to Google
The practical payoff is cheaper, faster training. In the filing's tests on a language benchmark, the 30-block version beat a 42-block standard model at the same size budget. In other words, you get better results by rewiring the model instead of piling on more blocks. The filing also says training took up to 3 times fewer steps to reach comparable quality.
The listed uses are broad: translation, speech recognition, code generation, image captioning, even robot control. No product is named. If a technique like this holds up in large models, the AI features you use could cost less compute to build, though the filing makes no promise about that.
Google's 50th filing we've tracked in our Enterprise AI coverage since May builds on ideas like keeping photos from adding fake details and keeping camera AI ready between shots.
Training big AI models burns enormous amounts of computer time and electricity, so any method that reaches the same quality in fewer steps has real money behind it. The filing claims up to 3 times fewer training steps, which is a large number aimed at a costly problem.
The fix is small compared with the problem. Each stage gets a few extra learned dials for deciding which earlier results to reuse, and the model starts out behaving like an ordinary one, so day one carries little risk. A small repair to how information travels through the model is a sensible match for the size of the issue.
The evidence is the open question. The numbers come from Google's own tests, with one language benchmark named. Whether the savings hold in the giant models that cost the most to train is what would decide if this matters.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
5 drawing sheets from US 2026/0330219 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in