Microsoft Files Patent for Sorting Screenshots by Topic Instead of Time
Your screenshot folder is probably a pile of files named by date and time. Microsoft's patent application describes an AI that sorts them by what is on screen, so a shopping spree and a work meeting stay apart even when you did both in the same hour.
What Microsoft's screenshot-sorting patent does
A screenshot folder fills up with files named only by date and time. Somewhere in there is the hotel you were comparing last week, but good luck finding it between a recipe, a work meeting and a receipt.
Microsoft's patent application describes a way to sort those screenshots by what they actually show. An AI reads the text on each one and looks at the pictures, then links screenshots that seem to be about the same thing. Each cluster of linked screenshots becomes a session, such as a shopping trip or a meeting.
Because the grouping follows meaning instead of the clock, it can handle your real habits: juggling two tasks at once, or coming back to a project days later. You could then search in plain language and get back the right pile of screenshots.
identifying one or more image regions and one or more text regions on the screenshot image, and generating features for the screenshot image based on the one or more image regions and the one or more text regions using an artificial intelligence (AI) module …
Translation: The system scans each picture to find visual elements and text so an AI model can understand them.
How the AI links screenshots into a web
The claims describe a pipeline that starts with a batch of screenshots. For each one, the system first splits the screen into image regions and text regions. The description says a screen region detection model (a neural network trained to spot where text, pictures and interface pieces sit) can do this.
Next, an AI module builds a set of features for each screenshot:
- Textual embeddings: optical character recognition (OCR, software that turns pictures of words into real text) reads the text regions, and a language model turns that text into numbers that capture its meaning.
- Visual embeddings: an image-analysis network turns the picture regions into numbers that capture layout, color and style.
- Optional extras: names of people and places, topics like travel or sports, and activity labels such as meeting, coding or browsing.
Then comes the user activity graph. Each screenshot is a node (a dot), and an edge (a line) joins two dots when their features look alike, measured by cosine similarity (a score for how closely two sets of numbers match). The filing also describes hyper-edges, which join three or more screenshots sharing a topic or name.
Finally, tightly connected clusters of dots are picked out as semantic sessions. A separate claim adds search: type a query, find the matching cluster, and get its screenshots back. Time can still count as one factor, but it no longer decides everything.
… generating a user activity graph comprising nodes and edges, wherein the nodes correspond to the plurality of screenshot images, and wherein the edges correspond to commonalities between node pairs based on the generated features …
Translation: It maps out your screenshots like a web of connected ideas by linking ones that share similar content.
Why your messy screenshot folder could change
If you screenshot everything, the practical payoff is finding things again. The filing says screenshots are usually stored with little structure, sorted by a filename with a timestamp, and that grouping by time alone lumps together tasks you did side by side. Under this approach, two tasks in the same hour could be split apart, and a project you return to days later could be joined back together.
The description says the system could run on your own device or on a server, and could also show you a visual map of your sessions or write short summaries of each one. It is a patent application, so no product is announced, and the description does not discuss privacy. Anything that reads the text on your screen raises the question of where that content goes and who can see it.
Microsoft's 27th filing in the AI vision work we've tracked since May builds on one about teaching objects without restarting and one on chained networks.
Claim 1 covers any computer system that takes a batch of screenshots, splits each into picture areas and text areas, and has an AI turn both into number-based fingerprints. It then links similar screenshots in a graph and pulls out clusters by meaning. The claim never names a specific AI model or says how alike two screenshots must be.
The tight spot is that both text and visual fingerprints are required, so a tool that grouped screenshots by text alone would fall outside claim 1 as written. Claim 20 is broader: it drops the region splitting and the fingerprints, and covers a graph of screenshots, a search query, a matching cluster and the results shown back to you.
If granted, claim 20 would have the most reach, because searching your screenshots by topic is the plainest use of the idea. The ingredients are all familiar, so any value sits in how they are combined.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Microsoft patent news breaks.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0311448 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in