Microsoft Patent: Two Coordinated Programs Help Computers Understand Object Locations

By Patentlyze Team · Updated Jul 24, 2026

What if you could ask an AI 'is there a clear path between the desk and the exit?' and it could actually look at a 3D map of the room and reason through the answer? That's the core idea behind this Microsoft patent.

What Microsoft's two-agent spatial AI actually does

Imagine asking a question like 'which objects in this warehouse are within two meters of the loading dock?' Today's AI chatbots would struggle with that because they're good at language but not at navigating structured data about physical spaces. Microsoft's patent describes a system designed to close that gap.

The trick is splitting the job between two AI agents that talk to each other. One agent (the 'reasoner') handles the big-picture thinking: it reads your question, figures out what information it needs, and sends a targeted request to its partner. The other agent (the 'retriever') translates that request into actual database code, queries a map or scene database, and sends the results back. The two keep passing information back and forth until the reasoner has enough to give you a real answer.

This back-and-forth loop means neither agent has to do everything on its own. The reasoner doesn't need to know how to write database queries, and the retriever doesn't need to understand the meaning of the question. Together, they can answer spatial questions that would trip up a single general-purpose AI.

How the reasoner and retriever agents divide the work

The patent describes a two-agent architecture built on top of large language models (AI systems trained on text) that is specifically designed for spatial reasoning tasks, meaning questions that require understanding where things are in physical space relative to each other.

The system works in a loop:

• The reasoner agent receives your question and a description of the scene's data structure (called a schema, basically a map of what fields and categories exist in the database). It uses a language model to figure out what to ask next.
• The retriever agent takes that sub-question and writes executable code, likely a database query in something like SQL or a graph query language, to pull the relevant spatial data from the scene database.
• The results come back to the reasoner, which either answers the original question or decides it needs more information and starts another loop.

The scene data could be anything from a 3D scan of a room (a format like a point cloud or scene graph) to a structured map of an industrial facility. The schema passed to both agents keeps them aligned on what data is actually available, so the retriever writes queries that will actually work.

The iterative loop is the key design choice here. Instead of asking one AI to do everything in one shot, the system refines its understanding step by step, which is a known technique for improving accuracy on complex reasoning tasks.

What this means for AI assistants in physical environments

Spatial reasoning is one of the messier unsolved problems in practical AI deployment. If you want an AI assistant to help navigate a physical environment, inspect a factory floor, or answer questions about a building layout, it needs to do more than chat. It needs to actually query structured spatial data and reason about the results. This patent suggests Microsoft is working on infrastructure to make that possible in a general way.

The most obvious connection is to Microsoft's growing investment in AI tools for enterprise and industrial settings, where questions about physical space come up constantly. Think warehouse logistics, facility management, or robotics. If this system works as described, it could make spatial queries as natural as asking a question in plain English.

Editorial commentary on a publicly published patent application. Not legal advice.