Get the newsletter
Salesforce · AI / ML · Filed Oct 23, 2025 · USPTO published Oct 1, 2026 · Oct 1 release

Salesforce's New Patent Grades AI Answers on Both Meaning and Exact Wording

Getting an AI to answer questions correctly is one thing. Getting it to answer them completely, in natural language, without missing key facts, is far harder. Salesforce has filed a patent on a scoring method that judges AI answers at two levels simultaneously, then feeds that grade back into training.

A person asks an AI agent a question about a network crash, and the AI agent provides an answer. Drawing from patent filing US 2026/0300631 A1.
A person asks an AI agent a question about a network crash, and the AI agent provides an answer.All 10 drawings ↓
Publication number
US 2026/0300631 A1(US20260300631A1)
Applicant
Salesforce, Inc.
Filed · Published
Oct 23, 2025 · Oct 1, 2026
Inventors
Shrikant Kendre, Juan Carlos Niebles Duque, Austin Xu, Shafiq Rayhan Joty, Honglu Zhou, Michael S. Ryoo
Status when we published
Waiting for an examiner
Document
20 claims
US classification
704/9
Parent application
Claims priority from a provisional application 63779512 (filed Mar 28, 2025)

How Salesforce grades AI answers to make them better

Every time a customer service chatbot gives you a wrong or half-baked answer, something failed in the way that AI was trained to know whether its responses were any good. Grading an AI's answers is surprisingly tricky: a response can sound correct without actually containing the right information, or it can include all the right facts but express them in a way that a simple comparison tool marks as wrong.

Salesforce's patent describes a system that grades AI answers on two levels at once. First, it checks whether the overall meaning of the answer matches a reference answer. Second, it checks whether the specific words and facts match. Each check produces a score, and the two scores are blended into a single grade that gets fed back to the AI so it can improve.

To make that first check work, the system first generates a "synthetic" reference answer, one that's worded the way a chatbot naturally talks, rather than a bare bullet-point fact. That way, the meaning-check doesn't unfairly penalize an AI for phrasing things conversationally. The end result is an AI that gets graded fairly and, over time, trained to be more accurate.

From the filing · CLAIM 1
… generating a first score indicative of a sentence-level semantic similarity between the synthetic answer and the generated response, based on a first similarity comparison between the embedding of the synthetic answer and the first embedding of the generated response; …

Translation: The system calculates how closely the AI answer matches the overall meaning of the target response.

How the two scores combine to train the language model

The system introduces a three-part evaluation loop for training large language models on question-answering tasks.

First, a small "synthesizer" model takes a ground-truth answer (think: the correct answer from a reference database) and rewrites it in the longer, more conversational style that a chatbot would naturally produce. This synthetic answer bridges the gap between a terse factual answer and the kind of fluent response a user would actually expect.

Second, an embedding model (software that converts text into numerical vectors representing meaning) encodes four pieces of text: the original ground-truth answer, the synthetic answer, and the AI's generated response (twice, using different encodings). From these, the system calculates two scores:

  • Sentence-level semantic score: how closely the overall meaning of the AI's response matches the synthetic answer, measured by comparing their vector representations.
  • Keyword-level score: how well the AI's response matches the ground-truth answer at the word and phrase level, using both exact matching and meaning-based keyword comparison.

Finally, the two scores are combined into a single reward signal. In reinforcement learning (a training technique where the model is rewarded for good outputs and penalized for poor ones, similar to how you'd train a dog with treats), this combined score tells the model how well it did and guides it to generate better answers in the next round.

From the filing · THE ABSTRACT
Embodiments described herein provide a QA framework that combines sentence-level semantic analysis with keyword-level semantic matching and/or exact keyword matching as feedback to iteratively update the generation and training process.

Translation: This method checks both the general meaning and specific words used to grade and improve the AI.

What better AI grading means for enterprise chatbots

For companies deploying AI assistants to answer customer questions, the quality of the answers depends almost entirely on how well the AI was trained to know when it got something right. Current scoring methods often fail at one extreme or the other: they either miss that an answer has the wrong meaning, or they penalize perfectly good answers that use different wording. A two-layer scoring approach could close both gaps at once.

Salesforce's run of LLM-training filings points to a company building the infrastructure to host and refine AI models for its enterprise clients in-house. If this scoring method works as described, it could make AI assistants in CRM and customer-service tools noticeably more reliable, without requiring a much larger or more expensive underlying model.

Salesforce files its 17th application we've tracked since May in the AI guardrails race, adding to earlier work like its agent permission system and its two-graph accuracy fix.

Source: US 2026/0300631 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.

Get our take in your Top Stories

Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Salesforce patent news breaks.

There are more where this came from

We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.

The drawings · 10 sheets

10 drawing sheets from US 2026/0300631 A1 · click any drawing to enlarge

User asks AI agent question., drawing sheet 1 from US 2026/0300631 A1
Sheet 1 User asks AI agent question.
AI grades ad description., drawing sheet 2 from US 2026/0300631 A1
Sheet 2 AI grades ad description.
AI answers video and document questions., drawing sheet 3 from US 2026/0300631 A1
Sheet 3 AI answers video and document questions.
AI answers image and text questions., drawing sheet 4 from US 2026/0300631 A1
Sheet 4 AI answers image and text questions.
Computer with question-answering components., drawing sheet 5 from US 2026/0300631 A1
Sheet 5 Computer with question-answering components.
Neural network for question answering., drawing sheet 6 from US 2026/0300631 A1
Sheet 6 Neural network for question answering.
Network of devices and servers., drawing sheet 7 from US 2026/0300631 A1
Sheet 7 Network of devices and servers.
Steps for grading AI answers., drawing sheet 8 from US 2026/0300631 A1
Sheet 8 Steps for grading AI answers.
Show the other 2 sheets
Salesforce’s New Patent Grades AI Answers on Both Meaning and Exact Wording patent drawing, sheet 9
Sheet 9
Salesforce’s New Patent Grades AI Answers on Both Meaning and Exact Wording patent drawing, sheet 10
Sheet 10
Patent filing page

More from Salesforce

All 110 Salesforce patents →

More in AI training & infrastructure

All 323 AI training & infrastructure patents →

Watchlists this filing belongs to

All watchlists →
Reader comments

No comments yet

Start the discussion

Real name or a handle, either is fine. Comments are read by a person before they appear, so allow a little time. Keep it about the filing.