Prophesee partners with Qualcomm on faster event-based smartphone cameras

Prophesee has partnered with Qualcomm to bring a new kind of fast image sensor to smartphone cameras.

Metavision’s “event-based” technology will enable smartphone cameras that can capture fast action compared to today’s image sensors. The company made the announcement with Qualcomm at the Mobile World Congress event in Barcelona.

The idea is to bring the speed, efficiency and quality of neuromorphic vision to mobile devices.
The technical and business collaboration will provide mobile device developers with a fast and efficient way
to leverage the ability of the Paris-based Prophesee sensor to dramatically improve camera performance, particularly in dynamic, fast-moving scenes (e.g. sports scenes) and in low light, through its innovative continuous and asynchronous pixel-based detection approach at events.

Unlike regular image sensors, the event-based sensors designed by Prophesee only record what changes in a smartphone image. This allows them to bypass the processing required by other kinds of sensors that process every pixel in an image. Prophesee only captures changes or events that reflect something changing in an animation.

Prophesee and Qualcomm have agreed to a multi-year partnership to enable native compatibility between Prophesee’s event-driven Metavision Sensors & Software and premium Snapdragon mobile platforms.

The world is neither raster nor frame based. Inspired by the human eye, Prophesee Event-Based sensors correct motion blur and other image quality artifacts caused by conventional sensors, especially in high-dynamic scenes and low-light conditions, bringing Photography and Video closer to our real-life experiences, Prophesee said.

“Prophesee is a clear leader in applying neuromorphic techniques to address the limitations of traditional
cameras and improve the overall user experience. We believe this is game-changing technology
taking mobile photography to the next level and our collaboration on both the technical and business fronts
The tiers will help drive adoption by leading OEMs,” Judd Heape, vice president of product management at Qualcomm Technologies, said in a statement. “Their breakthroughs with the shutterless capability of event cameras deliver a significant improvement in the quality of photography available in the next generation of Snapdragon-powered mobile devices, even in the most demanding environments, unlocking a range of new capabilities for Snapdragon customers. “

Prophesee’s neuromorphic sensors and Metavision event-driven software will be available for premium Snapdragon mobile platforms. Development kits are expected to be available from Prophesee this year.

“We are excited to partner with the provider of one of the most popular mobile platforms in the world
integrate event-driven vision into the Snapdragon ecosystem. Through this collaboration, product
Developers will be able to dramatically improve the user experience with imaging cameras
Quality and operational excellence are not available simply using traditional framework-based methods,” said Luca
Verre, CEO of Prophesee, in a statement.

How does it work

Prophesee’s innovative sensors add a new dimension of sensation to mobile photography. They change
the example in traditional image capture focusing only on changes in a scene, pixel by pixel,
continuously, at extreme speeds, the companies said.

Each pixel in the Metavision sensor incorporates a logic core, allowing it to function as a neuron. Each of them intelligently and asynchronously activates according to the number of photons it senses. A pixel that fires itself is called an event. Essentially, events are driven by the dynamics of the scene rather than an arbitrary clock anymore, so the capture speed always matches the actual dynamics of the scene.

High-performance event-based blurring is achieved by synchronizing a frame and Prophesee’s
event based sensor. The system then fills the gaps between and within frames with microsecond events to algorithmically extract clean motion information and correct for motion blur.

It is expected to be a development kit that will feature compatibility with Prophesee sensor technologies
available this year. Prophesee has more than 100 engineers and 50 international patents.