Researchers used deep learning techniques to enhance the image quality of a camera with a metalens integrated directly onto a CMOS imaging chip (left). The metalens manipulates light using an array of 1000-nm tall cylindrical silicon nitride nano-posts (right). (Image: Ji Chen, Southeast University)
Researchers have leveraged deep learning techniques to enhance the image quality of a metalens camera. The new approach uses artificial intelligence (AI) to turn low-quality images into high-quality ones, which could make these cameras viable for a multitude of imaging tasks including intricate microscopy applications and mobile devices.
Metalenses are ultrathin optical devices — often just a fraction of a millimeter thick — that use nanostructures to manipulate light. Although their small size could potentially enable extremely compact and lightweight cameras without traditional optical lenses, it has been difficult to achieve the necessary image quality with these optical components.
“Our technology allows our metalens-based devices to overcome the limitations of image quality,” said Research Team Leader Ji Chen, Ph.D., from Southeast University in China. “This advance will play an important role in the future development of highly portable consumer imaging electronics and can also be used in specialized imaging applications such as microscopy.”
The researchers describe how they used a type of machine learning known as a multi-scale convolutional neural network to improve resolution, contrast, and distortion in images from a small camera — about 3 cm × 3 cm × 0.5 cm — they created by directly integrating a metalens onto a CMOS imaging chip.
“Metalens-integrated cameras can be directly incorporated into the imaging modules of smartphones, where they could replace the traditional refractive bulk lenses,” said Chen. “They could also be used in devices such as drones, where the small size and lightweight camera would ensure imaging quality without compromising the drone’s mobility.”
Here is an exclusive Tech Briefs interview, edited for length and clarity, with Chen.
Tech Briefs: What was the biggest technical challenge you faced while leveraging deep learning techniques to enhance the image quality of a metalens camera?
Chen: The biggest challenge was that we are not computer science researchers and not the most knowledgeable in deep learning algorithms. As a result, we have gone through a long period of learning and experimentation to determine which neural network to use. Even now, we cannot guarantee that the neural network we are using is the best one. Therefore, it would be great if professionals in computer science could review our work and suggest more suitable neural network algorithms.
Tech Briefs: Can you explain in simple terms how everything works?
Chen: We generate many high- and low-quality imaging data pairs, and then let the neural network learn the features of these high- and low-quality images. The learned neural network then has the ability to process low-quality images into high-quality images. Therefore, when the device is used, as long as the low-quality imaging is taken and sent into the neural network for processing, high-quality imaging results can be obtained immediately.
Tech Briefs: The article I read says, “The researchers are now designing metalenses with complex functionalities — such as color or wide-angle imaging — and developing neural network methods for enhancing the imaging quality of these advanced metalenses.” How is that coming along? Any updates you can share?
Chen: Recently we have realized deep learning enhanced color super-resolution imaging by using an achromatic metalens. The schematic is shown below. The method used in this work is similar to the one in our article published in Optics Letters. However, here we can handle color images in a more complex manner. This requires our metalens to operate across a range of wavelengths, unlike the OL article, where it only worked at a single wavelength. We designed an achromatic meta-lens structure that can function across the 400-700 nm visible light spectrum, which naturally makes the lens structure more complex. Consequently, the neural network structure also needs to be more complex, processing the data for the RGB channels separately. Ultimately, we achieved very good results.
Figure 1. The overall process of our deep learning based super-resolution method for color imaging in metalens-integrated camera. (Image: The researchers)
Figure 2. Metalens-integrated camera and its architecture. (a) Photograph of the metalens-integrated camera. The round metalens area can be seen in the middle. (b) The architecture of the metalens-integrated camera and the color imaging process of a far-field object. The zoom-in picture is the detail of the image on CMOS sensor. Due to the limitation of the sensor’s pixel size, the image detail is difficult to image clearly. (c) Two types of metalens nanostructures, the hollow GaN nano-pillar (left) and the solid GaN nano-pillar. (d) The SEM image of the GaN metalens. (e) The focusing performance of the achromatic metalens, different wavelengths are focused at nearly the same position. (Image: The researchers)
Tech Briefs: Going from that, what are your next steps? Do you have plans for further work?
Chen: Next, we will conduct intelligent content analysis and recognition on the quality-enhanced images, such as identifying whether the images contain target objects and determining the orientation and position of these objects. This will be crucial for communication, positioning, and detection purposes. We are currently conducting research in this area. In addition, we are also attempting to apply this technology to microscopic imaging, aiming to identify the content of enhanced quality microscopic images, such as the recognition of pathological cells. This will play an important role in medical diagnostics as well.
Tech Briefs: The article also says, “To make this technology practical for commercial applications would require new assembly techniques for integrating metalenses into smartphone imaging modules and image quality enhancement software designed specifically for mobile phones.” Do you have any plans for that?
Chen: We are currently collaborating on a project with a domestic smartphone lens manufacturer in China. The project's main objective is to replace 1-2 refractive lenses in the smartphone lens module with metalenses and then comprehensively analyze the imaging performances of the new lens module. This project is currently underway.
Tech Briefs: Do you have any advice for researchers aiming to bring their ideas to fruition (broadly speaking)?
Chen: Here is some personal advice:
1) Define Clear Goals: Start with a clear vision and specific objectives. Understand what you want to achieve and break it down into manageable milestones.
2) Conduct Thorough Research: Before diving in, make sure you have a deep understanding of the field. This includes reviewing existing literature, identifying gaps, and understanding the current trends and technologies.
3) Build a Strong Network: Collaborate with others in your field. Networking can provide new perspectives, resources, and potential collaborators who can help advance your research.
4) Stay Persistent and Resilient: Research can be challenging and often involves setbacks. Stay persistent, learn from failures, and remain adaptable to overcome obstacles.
5) Focus on Practical Applications: Think about how your research can be applied in the real world. Practical applications can increase the impact of your work and attract more interest and support.
6) Keep Learning and Adapting: The field of research is always evolving. Stay updated with the latest developments, continuously learn new skills, and be open to adapting your approach based on new information.
Tech Briefs: Is there anything else you’d like to add that I didn’t touch upon?
Chen: The method of using neural networks to enhance the imaging quality of metalens cameras not only includes the approach proposed in our paper, which involves processing the imaging results. It also encompasses employing neural networks for the intelligent design of the microstructure of metalenses. This entails transforming the originally regular and periodic structure into non-periodic and arbitrarily shaped complex structures to improve the performance of metalenses and enrich their functionality.
Topics:
Artificial Intelligence Cameras Imaging Lenses Optical Components Optics Photonics