Google Patents a System That Picks the Right AI Model for Each Question
Not every question needs the most powerful AI in the building. Google has filed a patent for a system that figures out, question by question, which AI model is the right tool for the job, trading off speed against accuracy on the fly.
How Google routes your questions to the right AI
Every time you ask an AI assistant something, a server somewhere spends real money and real time generating your answer. A quick question like "What's the capital of France?" demands the same expensive machinery as "Explain the trade-offs in modern macroeconomic theory", even though it really doesn't need to.
Google's patent describes a system that learns to sort questions into two buckets: ones that a fast, cheaper AI model can handle just fine, and ones that need the bigger, slower, more thorough model. The system watches how well each model does on a question, then builds a small "routing" model that remembers those lessons and applies them to new questions as they arrive.
The practical benefit is that you get faster, cheaper answers on the easy stuff, while trickier questions still go to the heavy-duty model. And because the routing is learned rather than hand-coded, it adapts. The system also tracks a hidden cost: when a bad or vague answer forces you to ask a follow-up question, that's extra work for both you and the server. The routing model is trained to avoid that outcome, not just to be fast.
… determining, based on the corresponding measures, ground truth classification labels, each of the ground truth classification labels being for a corresponding one of the generative models; …
Translation: The system figures out the correct performance score for every AI model tested.
How the routing model learns which AI to trust
The patent describes a training pipeline for a model-routing classifier, a small, lightweight AI whose only job is to decide which of several larger AI models should answer a given question.
Here's how the training loop works:
- A question comes in and its features are extracted (things like length, topic signals, and linguistic patterns).
- The system sends that question to all the candidate AI models simultaneously and collects their outputs.
- Each output is evaluated to produce a quality score, which becomes a ground-truth label (essentially a grade: did this model do a good enough job?).
- Those labels, paired with the original question features, become a training example for the routing classifier.
Once trained, the routing classifier intercepts incoming questions and predicts, before any large model runs, which model is most likely to give a good answer at the lowest cost.
The system is designed around a specific insight: the cost of a bad answer isn't just one bad answer. When a response is incomplete or wrong, users send follow-up questions, which triggers another expensive model run. The routing model is trained to account for that downstream cost, not just the immediate one. This makes it meaningfully different from a simple "use the small model when the question is short" heuristic.
Those implementations reduce latency and/or conserve computational resource(s) through selection, for various requests, of a more computationally efficient generative model for utilization in lieu of a less computationally efficient generative model.
Translation: Choosing a smaller AI model saves computer power and speeds up replies when a massive model is not needed.
What this means for AI speed and answer quality
AI inference (the act of a model generating a response) is one of the largest and fastest-growing costs in cloud computing. Companies running AI at scale pay for compute by the second, and a system that correctly routes even 30-40% of queries to a cheaper model could translate to very large infrastructure savings without users noticing any drop in quality.
For end users, the benefit is latency: fast models respond in a fraction of the time of large ones. If the routing is accurate, your quick questions get near-instant answers, and the harder ones still get the full treatment. The patent's attention to follow-up questions is also telling, it frames bad AI answers not just as a quality problem but as a cost multiplier, which is the right way to think about it at Google's scale.
Google's 23rd filing we've tracked on AI models working together since May builds on earlier work like AI grading its own answers and AI grabbing the right tool.
The problem this patent attacks is real and expensive. Running a massive AI model on every single user query, regardless of complexity, is wasteful in a way that compounds fast, billions of queries a day times unnecessary compute adds up to serious money and carbon. That's not a niche engineering concern; it's one of the central economics of the AI industry right now.
The approach here is well-matched to the problem's size. Training a lightweight classifier on the observed outputs of heavier models is a principled method, and the decision to bake in the cost of follow-up questions is a genuinely useful framing. Most routing systems optimize for "did the model answer correctly?", this one also asks "did the answer avoid making things worse?"
The less exciting reality is that a growing pile of Google AI-efficiency filings suggests this is infrastructure plumbing, not a product announcement. The classifier itself is the novel piece, but the broader idea of "use a small model when you can, a big one when you must" has been a live engineering practice at every major AI lab for years. Whether Google's specific training recipe proves meaningfully better than existing approaches is a question the patent can't answer.
There are more where this came from
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings
6 drawing sheets from US 2026/0288789 A1 · click any drawing to enlarge
Want this weekly breakdown for a company we don't cover? Patentlyze Pro →
Be the first to weigh in