This ultracompact camera has optical elements that function as a neural network and can classify images at the speed of light. Researchers/Princeton Engineering
Researchers at Princeton University and the University of Washington made a breakthrough when they developed a camera the size of a salt grain – less than half a millimeter wide – capable of capturing unbelievably clear, full-color images with unprecedented detail.
Now, it appears, the team has pushed the boundaries once again by developing a new type of compact camera designed for computer vision, a type of AI that helps computers identify objects in pictures and videos.
The prototype, which introduces a new approach to computer vision, relies on light instead of electricity, consumes far less energy than a traditional computer, and identifies objects at the speed of light.
The camera significantly reduces power consumption and produces results hundreds of times faster.Credit: Ilya Chugunov, courtesy of Princeton University
“The question for me was always how can we use algorithms to sense and understand the world,” Felix Heide, PhD, an assistant professor of computer science at Princeton University and one of the study’s authors, says.
“This is a completely new way of thinking about optics, which is very different from traditional optics,” Arka Majumdar, PhD, a professor in electrical and computer engineering and physics at the University of Washington, and the study’s second author, adds. “It’s end-to-end design, where the optics are designed in conjunction with the computational block.
“Here, we replaced the camera lens with engineered optics, which allows us to put a lot of the computation into the optics,” Arka continues.
Delving into the research
Heide recalls that the idea took shape as he began exploring metasurfaces – artificial sheet-like materials with sub-wavelength features.
Due to their unique geometry, metasurfaces don’t bend light through glass or plastic like traditional lenses. Instead, they diffract light around tiny structures, much like how light spreads when passing through a narrow slit.
To build them, Heide and his students joined forces with the Washington Nanofabrication Laboratory experts, who specialize in ultra-small light-controlling devices, to engineer the camera and fabricate the chip.
But instead of using a traditional glass or plastic lens, the team integrated 50 stacked, flat and lightweight meta-lenses, which use microscopic nanostructures to manipulate light.
The optics in this camera relies on layers of 50 meta-lenses.Credit: Ilya Chugunov, courtesy of Princeton University
These meta-lenses double as an optical neural network, a brain-inspired AI system, giving the approach several key advantages. Despite being incredibly fast, identifying and classifying images over 200 times quicker than conventional neural networks, it’s also highly energy-efficient, relying on incoming light to operate instead of electricity.
In exploring this technology, Heide and his team were amazed to find that the light emerging from the pillar arrays didn’t need to resemble the original image at all. Instead, the pillars acted as specialized filters, sorting optical data into categories like edges, light and dark areas, or even features beyond human perception, thus providing computers with pre-processed, structured information.
“We realized we don’t need to record a perfect image,” Heide says. “We can record only certain features that we can then aggregate to perform tasks such as classification.”
Further insights
Before long, the team developed a system capable of identifying objects in images using less than 1% of the computing power required by conventional methods, with the metasurface lens handling an impressive 99.4% of the workload.
The system introduces a new paradigm, performing hundreds of millions of calculations (FLOPS) instantly. While traditional neural networks apply mathematical filters (kernels) to extract data, requiring numerous calculations even for a few pixels, this one naturally conducts complex filtering as light passes through, enabling a few large filters to analyze the entire image at once.
Tiny pillars within each metasurface lens reorganize and abstract light without electricity or active control. Heide and Ethan Tseng, a PhD student under his guidance are certain the design’s success relies on using fewer large optical kernels and seamlessly integrating hardware and software.
“Animal vision is joint between optical hardware and neural back-end processing,” Tseng reveals, citing examples like the mantis shrimp, dragonfly, and cuttlefish, which detect light polarization, something conventional optics cannot.
“There are animals that have more exotic vision than what we have, and we suspect that the hardware of their eyes is working together with their brain to perform various tasks,” he continues.
“This could potentially extend beyond image processing, and this is where we are just touching the tip of the iceberg,” Heide continues in a press release.
RECOMMENDED ARTICLES
“Right now, this optical computing system is a research prototype, and it works for one particular application, Majumdar explains in a statement, adding that they see its potential to revolutionize multiple technologies. “That, of course, remains to be seen, but here, we demonstrated the first step. And it is a big step forward compared to all other existing optical implementations of neural networks.”
The research has been published in the journal Science Advances.
0COMMENT
ABOUT THE EDITOR
Georgina Jedikovska Georgina Jedikovska, journalist, plant engineer, oenophile and foodie. Based in Skopje, North Macedonia. Holds an MSc. degree in Horticultural Engineering, with a specialization in viticulture and oenology. Loves travelling, exploring new cultures, a good read, great food and flavorful wines. Enjoys writing about archaeology, history, and environmental sciences.
