The image shows both color information from the color camera and events (blue and red dots) from the event camera generated by a pedestrian running. Credit: Robotics and Perception Group, University of Zurich
Artificial intelligence (AI) combined with a novel bio-inspired camera achieves 100-times faster detection of pedestrians and obstacles than current automotive cameras. This important step for computer vision and AI achieved by researchers of the University of Zurich can greatly improve the safety of automotive systems and self-driving cars.
It's every driver's nightmare: a pedestrian stepping out in front of the car seemingly out of nowhere, leaving only a fraction of a second to brake or steer the wheel and avoid the worst. Some cars now have camera systems that can alert the driver or activate emergency braking. But these systems are not yet fast or reliable enough, and they will need to improve dramatically if they are to be used in autonomous vehicles where there is no human behind the wheel.
Quicker detection using less computational power
Now, Daniel Gehrig and Davide Scaramuzza from the Department of Informatics at the University of Zurich (UZH) have combined a novel bio-inspired camera with AI to develop a system that can detect obstacles around a car much quicker than current systems and using less computational power. The study is published in Nature.
Most current cameras are frame-based, meaning they take snapshots at regular intervals. Those currently used for driver assistance on cars typically capture 30 to 50 frames per second and an artificial neural network can be trained to recognize objects in their images—pedestrians, bikes, and other cars.
"But if something happens during the 20 or 30 milliseconds between two snapshots, the camera may see it too late. The solution would be increasing the frame rate, but that translates into more data that needs to be processed in real-time and more computational power," says Gehrig, first author of the paper.
The image shows both color information from the color camera and events (blue and red dots) from the event camera; bounding boxes show the detection of cars by the algorithm. Credit: Robotics and Perception Group, University of Zurich
Combining the best of two camera types with AI
Event cameras are a recent innovation based on a different principle. Instead of a constant frame rate, they have smart pixels that record information every time they detect fast movements.
"This way, they have no blind spot between frames, which allows them to detect obstacles more quickly. They are also called neuromorphic cameras because they mimic how human eyes perceive images," says Scaramuzza, head of the Robotics and Perception Group. But they have their own shortcomings: they can miss things that move slowly and their images are not easily converted into the kind of data that is used to train the AI algorithm.
Gehrig and Scaramuzza came up with a hybrid system that combines the best of both worlds: It includes a standard camera that collects 20 images per second, a relatively low frame rate compared to the ones currently in use. Its images are processed by an AI system, called a convolutional neural network, that is trained to recognize cars or pedestrians.
The data from the event camera is coupled to a different type of AI system, called an asynchronous graph neural network, which is particularly apt for analyzing 3D data that change over time. Detections from the event camera are used to anticipate detections by the standard camera and also boost its performance.
"The result is a visual detector that can detect objects just as quickly as a standard camera taking 5,000 images per second would do but requires the same bandwidth as a standard 50-frame-per-second camera," says Gehrig.
One hundred times faster detections using less data
The team tested their system against the best cameras and visual algorithms currently on the automotive market, finding that it leads to one hundred times faster detections while reducing the amount of data that must be transmitted between the camera and the onboard computer as well as the computational power needed to process the images without affecting accuracy.
Crucially, the system can effectively detect cars and pedestrians that enter the field of view between two subsequent frames of the standard camera, providing additional safety for both the driver and traffic participants—which can make a huge difference, especially at high speeds.
According to the scientists, the method could be made even more powerful in the future by integrating cameras with LiDAR sensors, like the ones used on self-driving cars.
"Hybrid systems like this could be crucial to allow autonomous driving, guaranteeing safety without leading to a substantial growth of data and computational power," says Scaramuzza.
More information: Daniel Gehrig et al, Low Latency Automotive Vision with Event Cameras, Nature (2024). DOI: 10.1038/s41586-024-07409-w
Provided by University of Zurich
