Cornell researchers have developed an optical neural network (ONN) that can filter relevant information from a scene before the visual image is detected by a camera, a method that may make it possible to build faster, smaller and more energy-efficient image sensors.

In “Image Sensing with Multilayer, Nonlinear Optical Neural Networks,” published March 23 in Nature Photonics, researchers in the lab of Peter McMahon, assistant professor of applied and engineering physics in Cornell Engineering, have been able to demonstrate that ONN pre-processors can achieve compression ratios of up to 800-to-1 — the equivalent of compressing a 1,600-pixel input to just 2 pixels — while still enabling high accuracy across several representative computer-vision tasks.

Led by Tianyu Wang, an Eric and Wendy Schmidt AI in Science Postdoctoral Fellow, and doctoral student Mandar Sohoni, the researchers tested the ONN image sensor with machine-vision benchmarks, used it to classify cell images in flow cytometers, and further demonstrated its ability measure and identify objects in 3D scenes.

The difference between digital systems and an optical neural network is that with digital systems, images are first saved and then sent to a digital electronic processor that extracts information. Such electronic processing is power-consuming and, more importantly, requires far more time for the data to be processed and interpreted.

“Our setup uses an optical neural network, where the light coming into the sensor is first processed through a series of matrix-vector multiplications that compresses data to the minimum size needed — in this case, four pixels,” Wang said. “This is similar to how human vision works: We notice and remember the key features of what we see, but not all the unimportant details. By discarding irrelevant or redundant information, an ONN can quickly sort out important information, yielding a compressed representation of the original data, which may have a higher signal-to-noise ratio per camera pixel.”

The group also tested reconstructing the original image using the data generated by ONN encoders that were trained only to classify the image.

“The reconstructed images retained important features, suggesting that the compressed data contained more information than just the classification,” Wang said. “Although not perfect, this was an exciting result, because it suggests that with better training and improved models, the ONN could yield more accurate results.”

Wang and Sohoni believe their work could have practical applications in fields such as early cancer detection research, where cancer cells need to be isolated from millions or billions of other cells. Using flow cytometry, cells flow rapidly past a detector in a microfluidic flow channel. An ONN that has been trained to identify the physical characteristics of the cancer cells can rapidly detect and isolate those cells instantly.

“To generate a robust sample of cells that would hold up to statistical analysis, you need to process probably 100 million cells,” Sohoni said. “In this situation, the test is very specific, and an optical neural network can be trained to allow the detector to process those cells very quickly, which will generate a larger, better dataset.”

Sohoni said ONNs can also be useful in situations where very low-power sensing or computing is needed. For example, image sensing on a satellite in space would require a device that uses very little power. In this scenario, the ability of ONNs to compress spatial information can be combined with the ability of event cameras to compress temporal information, since the latter is only triggered when the input signal changes.

This work was supported by funding from NTT Research, the National Science Foundation, the Kavli Institute at Cornell, the David and Lucile Packard Foundation and the Canadian Institute for Advanced Research Quantum Information Science Program.

Diane Tessaglia-Hymes is a communications specialist for the College of Engineering.