Samsung Files Patent for Keeping AI Chatbots Quick When Many People Ask at Once
When a chatbot server runs out of memory, somebody's answer has to wait. Samsung's patent application lays out a cost formula for deciding whose.
What Samsung's AI request-juggling patent does
You type a question into an AI chatbot and the answer stalls halfway through a sentence. Somewhere, a server is juggling your request along with many others from strangers, and it has run out of room on its graphics chip to hold everyone's work at once.
Samsung's patent application describes a way to decide who gets bumped. The server moves a paused conversation's working notes out to slower, roomier memory, then brings them back when it's that person's turn. The key is picking the right conversation to pause, by weighing how long the move takes, how much it slows everyone else, and how much more room that conversation will need soon.
The moving also happens in the background while the chip keeps writing answers, so nobody sits waiting for the shuffle.
… performing swap cost-aware scheduling, wherein the generating of the response to each of the plurality of requests comprises generating tokens that constitute the response to each of the plurality of requests through a decoding operation managed by a scheduler …
Translation: The system manages heavy traffic by carefully calculating the cost of moving data in and out of memory.
How the server picks what to pause and resume
A chatbot server keeps each conversation's running context in a KV cache (a stored copy of the text so far, so the model doesn't re-read everything for every new word). That cache lives in blocks of graphics chip (GPU) memory, and the chip only has so many. The claims cover a scheduler that, at every step of generating tokens (the word-sized pieces that make up a reply), decides which paused request to bring back (swap in) and which running request to push out to ordinary CPU memory (swap out).
The decision runs in a set order:
- Pick a paused request to swap in, scored by how long the transfer takes, how long it has waited, its remaining processing time, and the latency target (the response time the service promises).
- Count the memory blocks needed for the running requests plus that one.
- If free blocks fall short, score each running request's swap-out cost: transfer delay, lost overall speed, and how much memory it will want later.
- Swap out the lowest-cost request, switch contexts, and keep generating tokens.
The weights on those three costs shift with how busy the server is and how full its memory is.
The dependent claims add extras. Swaps can run in the background, interleaved with token generation. Only part of a request may be swapped out, since blocks already copied to CPU memory can simply be dropped. And a new request can get a partial prefill (starting its prep work on just a slice) when only a few free blocks remain. The filing's example is a request needing five blocks when three are free.
… determining a request to be swapped out from among the requests being executed, based on the swap-out cost, switching contexts by performing swap-in and swap-out …
Translation: It decides which user tasks to temporarily pause to free up space for incoming questions.
What this means for waiting on AI answers
If you use AI chatbots, the practical stake is how long you wait when the service is busy. The filing says its goal is to cut delays and make better use of limited chip memory, two things every large chatbot service has to balance. A smarter way to decide whose work gets parked could mean fewer stalls mid-answer when many people are asking at once.
The other beneficiary is whoever runs the servers. GPU memory is one of the scarcest and priciest parts of serving an AI model, so squeezing more requests into the same hardware matters to the people paying for it. The filing describes this as server-side software and names no specific Samsung product, so treat it as infrastructure work for now.
Samsung's 61st filing we've tracked since June in the AI chip wars builds on ideas like keeping chips from going idle and moving data with light.
The problem here is real and expensive. AI chatbots run on chips with a fixed amount of fast memory, and every extra user competes for it. When memory runs short, someone's answer slows down. Anyone who has watched a chatbot crawl at a busy hour has felt the result.
Samsung's fix matches that problem well: a bookkeeping formula that weighs the price of every move before making it, instead of simply bumping whoever arrived last. It is a sensible, narrow answer to a narrow, costly problem, and it does not try to be more than that.
The text we read includes no measured results, so how much waiting it would save is unknown. As a document, it reads as competent server plumbing: useful if it works, and invisible to you if it does.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Samsung patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
14 drawing sheets from US 2026/0319838 A1 · click any drawing to enlarge
Show the other 6 sheets
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in