Get the newsletter
Nvidia · AI / ML · Filed Jul 27, 2026 · USPTO published Nov 19, 2026 · This week's release

Nvidia Files Patent for Photo-Describing AI Built From a Team of Specialist Models

Instead of training one giant AI to see and talk at the same time, Nvidia's filing hands the job to a team of separate specialists. One judges depth, one finds edges, one reads text, and one spots objects.

An AI combines clues from specialized vision tools to answer questions about a photo and describe what is happening. Drawing from patent filing US 2026/0369048 A1.
An AI combines clues from specialized vision tools to answer questions about a photo and describe what is happening.All 13 drawings ↓
Publication number
US 2026/0369048 A1(US20260369048A1)
Applicant
NVIDIA Corporation
Filed · Published
Jul 27, 2026 · Nov 19, 2026
Inventors
Animashree Anandkumar, Linxi Fan, Zhiding Yu, Chaowei Xiao, Shikun Liu
Status when we published
Waiting for an examiner
Document
20 claims
US classification
382/157
Parent application
Continuation of 18/544,840 (filed Dec 19, 2023)(18544840)

What Nvidia's team-of-specialists captioner does

Ever tried to learn piano, cooking and Spanish all in the same week? You'd probably get worse at all three. Many AI systems that describe photos are trained that way: one giant program learns to see, read and reason about pictures all at once, which takes enormous piles of images and computing time.

This Nvidia filing splits the job up. Existing specialist programs each look at the same photo and report what they're good at: how far away things are, where the edges sit, which objects and text show up. Those reports get squeezed into a short, fixed-size summary. A picture-reading part and a word-writing part then use it to produce a caption, or an answer to a question like "What's this person doing?"

The specialists stay as they are, so only a small slice of the system (about a fifth of its adjustable settings, per the filing) needs training. Nvidia reports results competitive with systems trained on far more images.

From the filing · CLAIM 1
… generating, using a vision encoder and language decoder and based on the image and the fixed length sequence of multi-modal features, a prediction of a textual description corresponding to the image.

Translation: Using visual and language components to write an accurate text description of the input image.

How the experts' labels get squeezed to a fixed size

The claims cover a three-step method. First, one or more pretrained neural networks (the "experts") look at an image and produce labels. These can be depth data, surface direction (called surface normals), edges, object labels, segmentation labels (which pixels belong to which thing), or text found in the image.

Second, a component the filing calls an experts resampler boils those labels down. The number of labels changes with how many experts are plugged in, so it uses cross-attention (a way for a fixed set of learned questions to look over a pile of inputs and pull out what matters). The result is a set number of tokens (small chunks of data the model reads). Add or remove an expert and the output size stays the same, which keeps memory use constant.

Third, a vision encoder and a language decoder take the image plus those tokens and write the text. That text can be a caption or an answer to a question. Small adaptors in each layer smooth the handoff from vision features to language features.

During training, the experts' weights are frozen (locked). Only the resampler and adaptors adjust, scored by comparing the model's text with a reference caption. The description says an expert can be swapped for a newer one with a light round of fine-tuning.

From the filing · THE ABSTRACT
Each expert is independently optimized for a specific task, facilitating the use of domain-specific data and architectures that are not feasible with a single large neural network trained for multiple tasks.

Translation: Smaller specialist models are trained separately on targeted data because one massive model is too inefficient.

Why cheaper training matters for photo-reading AI

If it works as described, building a system that can describe photos and answer questions about them gets cheaper. The filing reports competitive results with up to two orders of magnitude less training data, and only about 20% of the system's adjustable settings need training. That means fewer GPU hours and less image data to collect.

The filing lists uses from robots and self-driving perception to augmented reality and data centers. It also says the system holds up when some experts return pure noise, so you can add experts without hurting results. The accuracy gains from extra experts are small, though: 72.17% with none, about 72.8% with six, flattening near 73%.

This is the 17th Nvidia patent we've tracked since July in our AI models working as teams, following one on fixing floating 3D objects and one on a raw data translator.

Source: US 2026/0369048 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Nvidia patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 13 sheets

13 drawing sheets from US 2026/0369048 A1 · click any drawing to enlarge

Combining multiple image analyses to caption photos and answer questions, drawing sheet 1 from US 2026/0369048 A1
Sheet 1 Combining multiple image analyses to caption photos and answer questions
Feeding expert visual clues into text generators during AI training, drawing sheet 2 from US 2026/0369048 A1
Sheet 2 Feeding expert visual clues into text generators during AI training
Blending clues from specialist models into a standard format, drawing sheet 3 from US 2026/0369048 A1
Sheet 3 Blending clues from specialist models into a standard format
Converting visual details into plain words and sentences, drawing sheet 4 from US 2026/0369048 A1
Sheet 4 Converting visual details into plain words and sentences
Tuning the visual system to handle new tasks efficiently, drawing sheet 5 from US 2026/0369048 A1
Sheet 5 Tuning the visual system to handle new tasks efficiently
Steps for turning specialist vision outputs into written answers, drawing sheet 6 from US 2026/0369048 A1
Sheet 6 Steps for turning specialist vision outputs into written answers
Test scores showing accuracy gains from adding more visual experts, drawing sheet 7 from US 2026/0369048 A1
Sheet 7 Test scores showing accuracy gains from adding more visual experts
Comparing model accuracy when adding depth clues and noisy data, drawing sheet 8 from US 2026/0369048 A1
Sheet 8 Comparing model accuracy when adding depth clues and noisy data
Show the other 5 sheets
Inside the graphics chip that runs the AI models, drawing sheet 9 from US 2026/0369048 A1
Sheet 9 Inside the graphics chip that runs the AI models
Connecting multiple graphics chips directly together to share memory, drawing sheet 10 from US 2026/0369048 A1
Sheet 10 Connecting multiple graphics chips directly together to share memory
Main computer parts linking processors to screens and inputs, drawing sheet 11 from US 2026/0369048 A1
Sheet 11 Main computer parts linking processors to screens and inputs
How cloud servers train AI and send answers to apps, drawing sheet 12 from US 2026/0369048 A1
Sheet 12 How cloud servers train AI and send answers to apps
Streaming video and receiving user commands over a remote network, drawing sheet 13 from US 2026/0369048 A1
Sheet 13 Streaming video and receiving user commands over a remote network
Patent filing page

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.