Samsung Patent: Loading Only Required AI Model Sections Per Token Reduces Latency

By Patentlyze Team · Updated Jul 3, 2026

Running a large AI model on a phone or PC means constantly shuffling massive amounts of data in and out of memory. Samsung's new patent describes a way to predict which pieces of the model you'll need next and load them early, before the AI even asks.

How Samsung's selective AI pre-loading works in plain terms

Imagine a massive cookbook stored in a warehouse. Every time you want to cook something, a worker has to run to the warehouse, find the right page, and bring it back before you can start. Now imagine if a helper could predict which pages you'll need and bring them to your kitchen counter in advance. That's the core idea here.

Large AI models like the ones that power chatbots are too big to fit entirely in fast memory at once. When the AI processes each word in your prompt, it has to fetch the relevant sections of the model from slower storage. Samsung's patent describes a system that uses a smaller, faster AI to predict which sections of the big model will be needed for the next word, and loads them into fast memory ahead of time.

The result is that the main AI model spends less time waiting around for data to arrive. On a device with limited memory, like a smartphone or a laptop, that kind of advance planning can make a real difference in how quickly you get a response.

How the system picks and loads subnetworks per token

The patent describes a prefetching system (pre-loading data before it's requested) for generative large language models. When an AI processes text, it moves through the model layer by layer for each token (a token is roughly a word or word-fragment). Each layer contains many neurons, but not all of them are equally relevant for every input.

Samsung's approach involves a secondary machine learning model that watches the incoming token and predicts which subnetworks (specific subsets of neurons within each layer) the main model will actually activate. Before the main model even gets there, those subnetworks are written into fast-access memory.

• A token arrives for the AI to process
• A lightweight predictor model identifies the relevant subnetworks across multiple layers
• Those subnetworks are written to memory in advance
• The main model runs using only those pre-loaded subnetworks

This is related to research on sparse activation, the observation that large models don't use all their neurons for every input. By loading only what's needed, the system reduces memory bandwidth pressure, which is often the main bottleneck when running large AI models on consumer hardware.

What this means for AI on Samsung phones and devices

Running AI models locally on phones and laptops (rather than in the cloud) is a major push across the industry right now. The main obstacle is that these models are enormous, and device memory is limited and slow compared to a data center. A system that accurately pre-loads only the relevant model fragments could make on-device AI meaningfully faster without requiring more memory or a bigger chip.

For Samsung, which sells both the Galaxy smartphone line and the Exynos chips inside them, this kind of optimization is directly applicable to its own hardware. If the prefetching predictor is accurate enough, users could see faster AI response times in features like on-device translation, writing assistance, or future Galaxy AI capabilities without any change to the underlying model size.

Editorial commentary on a publicly published patent application. Not legal advice.