OpenAI Files Patent for Keeping Thousands of Custom AI Models Ready to Answer
When every customer has their own tuned version of an AI model, someone has to decide which versions stay ready and which get shelved. OpenAI's patent application describes a three-tier shelf system, and a line-sorting rule that serves already-loaded models first.
What OpenAI's custom-model switching setup does
Ever stood in a busy sandwich shop where every regular has a slightly tweaked order? The kitchen is the same for everyone, but somebody has to keep track of whose tweaks go where. That is the puzzle OpenAI describes for AI services that give each customer their own custom version of a big model.
OpenAI's patent application sorts the line. Customers whose custom version is already loaded and ready get served first. Everyone else waits a moment while their version is fetched. Versions used recently stay close at hand, and rarely used ones get shelved in a bigger, slower storage room.
The goal is to serve a huge number of customers at once without keeping every custom version loaded all the time. Your request shouldn't sit around while the computer shuffles files.
… organizing the plurality of requests into first requests and second requests based on which requests of the plurality of requests use currently-loaded fine-tuned models that are on one or more inference engines of the generative response system when the plurality of requests are received …
Translation: The system sorts incoming tasks depending on whether the needed AI personality is already sitting in active memory.
How OpenAI decides which model stays loaded
The filing describes a generative response system: a group of servers (called inference engines) that run a large language model. Instead of building a whole new model for each customer, it uses small add-ons called LoRAs (low-rank adapters, compact sets of numbers that nudge the big base model toward a customer's needs). One add-on can be over a thousand times smaller than the base model, so a single engine can hold hundreds.
A controller sorts incoming requests into two groups, which is what claim 1 covers. Requests for add-ons that are already loaded go first, and the rest follow once their add-ons are in place. Requests for the same add-on are batched and run side by side, even if they arrive at different times.
Add-ons live in a three-tier cache (a nearby holding area for things you may need again):
- GPU memory: the fastest tier. The example holds about 500 add-ons, ready instantly.
- CPU RAM: the server's main memory. The example holds about 2,000, and loading from here takes roughly 50 milliseconds.
- Network storage: can hold millions. Fetching from here takes about 5 seconds in the example.
When space runs out, the longest-unused add-on is swapped out, and a copy stays in RAM. Other rules keep rarely used add-ons from waiting forever, spread requests across engines, and shift memory between add-on storage and the working notes (the KV cache) a model keeps while writing a reply.
Requests for a model that is not currently loaded are processed after swapping the model onto an engine, according to a caching hierarchy, to replace a long-dormant (or infrequently used) model.
Translation: If an unready AI model is requested, the system swaps it into memory by kicking out an older unused one.
Why this matters for businesses with custom AI
If you or your company pay to customize an AI model, the price and speed you get depend on how cheaply the provider can keep that model on call. The filing says customers shouldn't have to book or pay for a reserved time slot. They can make a request whenever they like, and the system loads their model just in time and keeps it around until it has sat unused for a long while.
The filing also mentions a Fine-Tuning API, the tool developers use to customize models with their own data. This patent application describes the behind-the-scenes plumbing for a service like that. The text doesn't promise lower prices or faster answers, but the efficiency problem it targets (idle, underused GPUs) is a cost problem at its core.
OpenAI has filed its third patent in the topic we cover since September, following customers training with photos and a watchdog for models.
The idea is a sorting rule, and it runs entirely in software. A company that already operates a large fleet of AI servers could add this scheduling layer without any new chips or devices. That puts it much closer to a working feature than most hardware-heavy filings.
What has to exist first is the service it sits inside: a catalog of customer-tuned models and servers to host them. The filing's numbers (500 models in the fastest memory, 2,000 in the next tier, millions in storage) read like a description of a service with many customers.
You would never see this on a screen. The shortest route to a product is that it simply runs behind the curtain, and you notice faster replies or cheaper custom-model plans. Claim 1 itself is short and plain: serve what is already loaded first. The rest of the filing is careful bookkeeping around that rule.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time OpenAI patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0310938 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in