Nvidia Patents a Non-Linear Projection System for Rendering 3D Gaussian Scenes

By Patentlyze Team · Updated May 22, 2026

3D Gaussian Splatting is the hottest technique in real-time scene reconstruction right now — but it breaks down when cameras have fisheye or other non-standard lenses. Nvidia's new patent proposes a fix that could make Gaussian-based rendering work cleanly for exactly those tricky optics.

How Nvidia renders 3D scenes through distorted camera lenses

Imagine you're trying to reconstruct a 3D scene — a room, a street, a product shot — from a set of photos. One popular modern approach represents everything in the scene as millions of fuzzy 3D blobs called Gaussian particles. Your rendering engine figures out which blobs land on each pixel of your final image, blends them together, and out pops a photorealistic view. It works beautifully — until your camera has a wide-angle, fisheye, or otherwise distorted lens, where the standard math breaks down.

Nvidia's patent describes a system that handles those distorted cameras gracefully. Instead of projecting each 3D blob onto the camera plane with simple linear math (which assumes a perfectly standard lens), it uses a statistical trick called an unscented transform to sample a handful of representative points from each blob and project those through the actual, non-linear camera model. That way the system can still figure out which blobs are relevant to each pixel, even with a wonky lens.

Once the right blobs are identified, the system fires a ray through each pixel, finds the spot along that ray where each blob contributes the most, and blends all those contributions into a final color. The result: accurate, artifact-free renders from cameras that previously caused headaches.

How sigma points and ray intersection drive the pixel output

The patent describes a pipeline with several distinct stages.

First, a non-linear sigma-point projection step replaces the usual linear Gaussian projection. The unscented transform (a technique borrowed from estimation theory — think of it as a smarter way to push probability distributions through non-linear math without full Monte Carlo sampling) generates a small set of sigma points that characterize each 3D Gaussian blob. Those sigma points are then projected through whatever camera model is in use — including fisheye or other radially distorted lenses — to get a 2D footprint on the image plane.

Second, a particle filter uses those 2D footprints to decide which 3D Gaussian particles could plausibly affect each pixel. This is a culling step: most of the scene's millions of blobs are irrelevant to any given pixel, so skipping them saves enormous compute.

Third, for the surviving candidates, the system performs ray-based intersection: a ray is cast through each pixel back into 3D space, and the system finds the segment where that ray passes through each relevant Gaussian particle. Along that segment it computes a maximum-response point — the location where the Gaussian blob contributes most strongly.

• Project sigma points through non-linear camera model
• Filter particles by projected 2D overlap
• Trace rays and find peak contribution along each intersection
• Blend all contributions into a final pixel color

The output is an image that respects the actual optics of whatever camera captured the original scene data.

What this means for AI-generated 3D content and robotics

3D Gaussian Splatting has exploded in popularity for neural scene reconstruction — it's faster and often higher-quality than NeRFs for many tasks, and it's increasingly used in robotics simulation, autonomous vehicle sensor modeling, and AR/VR content pipelines. But almost all production-grade cameras — robot-mounted wide-angle cameras, automotive surround-view systems, VR headset cameras — have non-standard optics that the standard Gaussian Splatting math doesn't handle well. This patent directly targets that gap.

For Nvidia, whose Omniverse and robotics simulation platforms depend heavily on accurate sensor modeling, getting Gaussian Splatting to work correctly with real-world camera rigs is genuinely useful engineering. If this approach ships in their tooling, it means developers can feed in raw footage from fisheye or wide-angle cameras without pre-processing the distortion out — a meaningful workflow improvement.

Editorial commentary on a publicly published patent application. Not legal advice.