Nvidia Files Patent for Photo-Describing AI Built From a Team of Specialist Models
Instead of training one giant AI to see and talk at the same time, Nvidia's filing hands the job to a team of separate specialists. One judges depth, one finds edges, one reads text, and one spots objects.

- Publication number
- US 2026/0369048 A1(US20260369048A1)
- Applicant
- NVIDIA Corporation
- Filed · Published
- Jul 27, 2026 · Nov 19, 2026
- Inventors
- Animashree Anandkumar, Linxi Fan, Zhiding Yu, Chaowei Xiao, Shikun Liu
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 382/157
- Parent application
- Continuation of 18/544,840 (filed Dec 19, 2023)(18544840)
What Nvidia's team-of-specialists captioner does
Ever tried to learn piano, cooking and Spanish all in the same week? You'd probably get worse at all three. Many AI systems that describe photos are trained that way: one giant program learns to see, read and reason about pictures all at once, which takes enormous piles of images and computing time.
This Nvidia filing splits the job up. Existing specialist programs each look at the same photo and report what they're good at: how far away things are, where the edges sit, which objects and text show up. Those reports get squeezed into a short, fixed-size summary. A picture-reading part and a word-writing part then use it to produce a caption, or an answer to a question like "What's this person doing?"
The specialists stay as they are, so only a small slice of the system (about a fifth of its adjustable settings, per the filing) needs training. Nvidia reports results competitive with systems trained on far more images.
… generating, using a vision encoder and language decoder and based on the image and the fixed length sequence of multi-modal features, a prediction of a textual description corresponding to the image.
Translation: Using visual and language components to write an accurate text description of the input image.
How the experts' labels get squeezed to a fixed size
The claims cover a three-step method. First, one or more pretrained neural networks (the "experts") look at an image and produce labels. These can be depth data, surface direction (called surface normals), edges, object labels, segmentation labels (which pixels belong to which thing), or text found in the image.
Second, a component the filing calls an experts resampler boils those labels down. The number of labels changes with how many experts are plugged in, so it uses cross-attention (a way for a fixed set of learned questions to look over a pile of inputs and pull out what matters). The result is a set number of tokens (small chunks of data the model reads). Add or remove an expert and the output size stays the same, which keeps memory use constant.
Third, a vision encoder and a language decoder take the image plus those tokens and write the text. That text can be a caption or an answer to a question. Small adaptors in each layer smooth the handoff from vision features to language features.
During training, the experts' weights are frozen (locked). Only the resampler and adaptors adjust, scored by comparing the model's text with a reference caption. The description says an expert can be swapped for a newer one with a light round of fine-tuning.
Each expert is independently optimized for a specific task, facilitating the use of domain-specific data and architectures that are not feasible with a single large neural network trained for multiple tasks.
Translation: Smaller specialist models are trained separately on targeted data because one massive model is too inefficient.
Why cheaper training matters for photo-reading AI
If it works as described, building a system that can describe photos and answer questions about them gets cheaper. The filing reports competitive results with up to two orders of magnitude less training data, and only about 20% of the system's adjustable settings need training. That means fewer GPU hours and less image data to collect.
The filing lists uses from robots and self-driving perception to augmented reality and data centers. It also says the system holds up when some experts return pure noise, so you can add experts without hurting results. The accuracy gains from extra experts are small, though: 72.17% with none, about 72.8% with six, flattening near 73%.
This is the 17th Nvidia patent we've tracked since July in our AI models working as teams, following one on fixing floating 3D objects and one on a raw data translator.
Source: US 2026/0369048 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 13 sheets
13 drawing sheets from US 2026/0369048 A1 · click any drawing to enlarge
Show the other 5 sheets

No comments yet