Microsoft Patents a Way to Run Giant AI Models on Chips Too Small to Hold Them
Most AI models are too big to fit on the chips in edge devices and phones. Microsoft's new patent describes a server-assisted streaming system that sends an AI model to a device in bite-sized pieces, so the chip only ever has to hold a small slice at a time.
How Microsoft fits a big AI into a tiny chip's memory
Ever tried to stream a movie on a slow connection and watched it buffer endlessly? Running a large AI model on a small chip has a similar problem: the model is far bigger than the chip's memory, so it simply won't fit.
Microsoft's approach is to keep the full AI model on a separate server and send it to the device in chunks. The chip processes one chunk, hands back the results, and then receives the next chunk. To keep things moving, the system also splits the incoming data into very small groups (called microbatches) and adjusts their size automatically so the chip stays busy instead of waiting around for the next piece to arrive.
The practical result: a device with a modest chip could train or run a large AI model that would normally require a server room's worth of hardware. You wouldn't need to buy more expensive hardware; the server does the heavy lifting in coordination with whatever device you already have.
… at least one target device comprising an integrated circuit chip having an on-chip memory of a size less than an entirety of the AI model …
Translation: One of the target chips has less memory than the total size of the AI model.
How the server streams model pieces to the target chip
The patent describes a system with two main actors: a parameter server (a powerful machine that stores the complete AI model) and one or more target devices (chips or edge hardware whose on-chip memory is smaller than the full model).
The server's data manager holds the master copy of the model. A batch manager figures out the right data chunk size for each connected device based on how much memory that device has. The server then streams a portion of the model, sized to fit the target chip, and the chip begins processing.
Here's the clever overlap: while the chip is busy running one slice of the model, the server is simultaneously doing two other things:
- Updating the weights (the numerical values the AI has learned) for a different slice that already finished processing
- Sending the next slice to the device so it arrives just in time
This pipelining approach (keeping multiple tasks in flight at once, so nothing sits idle) is how the system avoids the obvious bottleneck: every time the chip finishes one piece, it would normally have to wait for the next one to download. By pre-fetching and overlapping the update work, the patent claims to cut that waiting time significantly. The microbatch size can also be tuned manually or automatically to find the sweet spot between download time and processing time.
The AI model may be dissected into smaller portions (e.g., layers or sub-layers), and each portion may be executed as efficiently as possible on the target device.
Translation: The AI model is chopped into smaller layers so the tiny chip can handle them.
What this means for AI training outside data centers
The immediate audience for this technology is AI researchers and companies that want to train or run large models on hardware that isn't a data center. Edge devices, specialized accelerator chips, and on-device AI are all areas where memory is tightly constrained and the models keep getting bigger.
For everyday users, the longer-term implication is that your device, whether a laptop, a dedicated AI accelerator, or eventually a phone, might one day participate in running or training models that currently require cloud infrastructure. That could mean faster, more private AI processing closer to you, without every computation needing to travel to a remote server. The design also leaves room for multiple devices to work together on a single model, which points toward distributed, cooperative AI workloads.
Microsoft's 15th filing we've tracked in the AI chip wars since July builds on ideas like splitting work across chips and routing training data inside data centers.
Streaming a large AI model in pieces solves the memory problem but creates a permanent dependency: the device can only work if its connection to the remote server stays fast and stable. On a flaky network, the whole system stalls.
The automatic chunk-size adjustment is sensible on paper, but it can behave unpredictably on hardware the tuner was never designed for, and more smaller chunks means more back-and-forth with the server.
The trade reads as worth it. AI models are growing faster than the memory budgets of everyday devices, and this approach gives those devices a practical path to run heavyweight software today rather than waiting for hardware to catch up.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
8 drawing sheets from US 2026/0288469 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in