Google Files Patent for Swapping a Bigger Language AI Into a Photo-Reading One
Rather than training an AI that can both see and talk from scratch, Google describes taking a finished image-reading model, swapping in a much bigger language model, and retraining only a small slice of the result.
What Google's glued vision-language model does
Every time you ask an AI to describe a photo, two skills have to work together: seeing the picture and finding the words for it. Google's filing is about building that pairing without starting from zero.
Image-reading AI often comes with a small language component built in. Google's idea is to keep the image-reading half, pull out that small language piece, and plug in a much bigger, already-trained language AI in its place. Two finished tools get glued into one.
Then comes the efficient part. Instead of retraining everything, the system adjusts only a small slice of the combined model, using pairs of pictures and matching text, and leaves the rest alone. The filing says that saves computing power and time.
… generating a glued neural network that includes parameters taken from both the visual encoder neural network and the large-scale language model neural network; …
Translation: Combining the image-reading components and the advanced language brain into a single hybrid system.
How only a slice of the model gets retrained
The claims describe a three-stage recipe. First comes upstream training. A multi-modal network (one that handles more than one kind of data, here images and text) is trained with a visual encoder (the part that turns a picture into numbers an AI can work with) and a small language model. Separately, a large language model is trained on text alone.
Second is the midstream stage, where the glued network is built. It runs the visual encoder, then an attentional pooling layer (a step that uses attention, a way of weighing which parts of the image matter most, to condense the encoder's output), then the layers of the large language model. Text is converted into embeddings (lists of numbers) and fed in alongside the image.
Training on this glued network is deliberately narrow:
- Only a proper subset of the parameters (the adjustable numbers inside the AI) gets changed, such as the pooling layer and optionally some layers of the language model.
- Everything else is held fixed.
- Training uses image and text pairs, scored by how far the predicted text is from the real text.
Third is downstream adaptation for a new task. That can mean zero-shot (no extra training), few-shot (a handful of examples), or fine-tuning. One described option runs trial runs that freeze different parts, then uses the performance gains to pick which parts to freeze.
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating and training a glued neural network that reuses components taken from a first pre-trained neural network and a second pre-trained neural network.
Translation: Google's patent details software techniques for recycling and merging different artificial intelligence models together.
Why reusing finished AI parts cuts the cost
The practical payoff is cost and speed. Adding image skills to a language model you already have normally means a huge new training run. This recipe reuses finished parts and retrains only a small slice, which the filing says can match or beat conventionally trained models while using less computing and less wall-clock time.
The description lists plenty of uses: answering questions about a photo, captioning, searching images by text description, controlling robots from camera views and instructions, and even health predictions from mixed patient data. It also cites Google's PaLM 2 language model and CoCa image-text model as example building blocks. You would never see this step in an app. You would see its effects in how quickly image abilities show up in AI tools.
Google's 100th filing we've tracked in Language AI since May follows a two-model search ranking system and an AI search training method.
Building a big AI model from scratch is one of the costliest jobs in software. Language models with tens or hundreds of billions of parameters take enormous amounts of computing to train, and teaching one to also see adds to that bill. Reusing finished parts goes after a real expense.
The approach fits the size of the problem. Freeze most of the combined model, adjust a small slice, and you spend a fraction of the effort. The filing says the result can match or beat conventionally trained models, though the text gives no numbers to check that.
You will never see this step in an app. It is a training recipe, so its value shows up as how cheaply a company can add image skills to a language model it already owns. That makes it useful inside Google's labs and quiet on your screen.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Google patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
5 drawing sheets from US 2026/0311476 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in