Sony Patents an AI Game Soundtrack That Reacts to Your Heart Rate
Sony's patent application describes a game soundtrack that an AI writes as you play, steered by readings like your heart rate, sweating and blinking. Your pulse becomes part of the score.

- Publication number
- US 2026/0364456 A1(US20260364456A1)
- Applicant
- Sony Interactive Entertainment Inc.
- Filed · Published
- Jul 23, 2026 · Nov 19, 2026
- Inventors
- Daniel Israel Pereira Jimenez, Estefania Rodriguez Quiros, Lazaros Michailidis, Jesus Barcias, Christopher William Henderson
- Status when we published
- Waiting for an examiner
- Document
- 20 claims
- US classification
- 463/35
- Parent application
- Continuation of 18/443,515 (filed Feb 16, 2024)(18443515)
What Sony's heart-rate game soundtrack does
You're creeping through a dark cave in a game, heart pounding, and the music is the same loop you heard an hour ago. It has no idea you're nervous. Sony's new patent application is about fixing that gap.
Here's the idea. Your console watches what's happening and picks a few mood keywords from a list the game's developer wrote, words like 'cave,' 'hostile,' or 'friendly.' It sends those words to an AI that composes music on the fly and keeps reshaping it as the words change.
The filing also has the keywords respond to your body. Sensors in a VR headset or controller can pick up your heart rate, sweating, or blinking, so the score can shift when you tense up. If a jump scare hits, the music can lurch with it.
… receive, from the entertainment device during processing of the game, one or more keywords selected based at least in part on physiological data of a user playing the game; and in response to receipt of a keyword of the one or more keywords, update the generation of the information indicative of the musical score based at least in part on the keyword.
Translation: The system listens to your body signals while you play and changes the music based on what it detects.
How keywords steer the AI composer mid-game
The claims cover a music generator that sits between the game and your speakers. When the console asks for a score, the generator starts producing music right away and streams it back as it goes. It can run on a cloud server, on a nearby device, or inside the console itself.
The key trick is the keywords. The console reads what the player is doing, including physiological data (body readings such as heart rate, eye movement, sweating, skin temperature and blinking), and picks matching words from a list the developer defined for that game. Each time a new keyword arrives, the generator updates the music. Sending a few words is far lighter than streaming the whole game state, and the filing says it also keeps the AI easier to train.
The filing describes several add-ons:
- Seed values: extra inputs, such as a game title or in-game location, that make the same keywords produce music unique to that game or place.
- Sudden events: a jump scare can trigger an abrupt change, or stop the AI so prerecorded audio plays instead.
- Smooth transitions: the AI can be trained on keyword pairs like 'meadow' then 'cave' so the music blends between moods.
- Feedback: players can say whether the music matched their mood, and that answer helps retrain the AI.
The output can be a MIDI file or stream, which is a set of note instructions rather than a finished recording.
The one or more keywords are indicative of one or more of an action and/or a condition associated with a user playing the game and the game state.
Translation: Special labels describe what the player is doing and how the game is reacting at that exact moment.
Why a reactive soundtrack matters for horror games
For players, the pitch is a soundtrack that fits how you play, so two people in the same level could hear different music. Today's game scores mostly play prerecorded tracks in a set order, and the filing says even the real-time approaches are limited by how many sound snippets a game ships with. An AI composer has no such ceiling.
For developers, the keyword list is the control panel. They decide which moods exist and which game moments trigger them, and seed values keep a game's music distinct from other games using the same AI. The filing's examples use a PlayStation 5, a VR headset and a DualSense controller. One tradeoff: a game would be reading body signals like your heart rate, and some players will want to think about that.
This is the 177th Sony filing we've tracked in our PlayStation strategy watch since May, adding to work like the cloud connection test game and the AI play guide ghost.
Source: US 2026/0364456 A1 at the USPTO (PDF). Full patent text and figures are in the official publication.
Get our take in your Top Stories
Liked this breakdown? Add Patentlyze as a preferred source on Google, and our plain-English take shows up more often in your Top Stories the next time Sony patent news breaks.
We read every patent application Big Tech publishes and send you the ones worth knowing. Plain English, free, every week.
The drawings · 5 sheets
5 drawing sheets from US 2026/0364456 A1 · click any drawing to enlarge

No comments yet