A new artificial intelligence system allowing shoppers on Facebook to identify characteristics of items in uploaded photographs is based on Cornell computer vision research into fine-grained visual recognition.

Announced May 19 by CEO Mark Zuckerberg via Facebook Live, the product-recognition system can identify attributes across billions of photographs in dozens of categories including fashion and home decor.

The “click photo and search” technology behind the system was first developed in 2015 by Kavita Bala, professor and chair of computer science, and her then-doctoral student, Sean Bell, M.S. ’15, Ph.D. ’16, now a research scientist at Facebook.

In 2016 Bala and Bell co-founded GrokStyle, a startup based on this research. GrokStyle was acquired by Facebook last year.

“This was an idea that came from a research setting in academia, with a student and the adviser,” said Bala, incoming dean of the Faculty of Computing and Information Science, beginning Aug. 15. “I am delighted to see it go all the way through to consumers in the world.”

The Facebook system, known as GrokNet and led by Bell, builds off the GrokStyle technology and automatically suggests attributes such as colors and materials when sellers upload photos of products for sale, according to Facebook. The technology allows buyers to conduct detailed searches when looking for specific items, even if the sellers didn’t add those details to the descriptions when they posted the photos.

“When you take a picture of something, we can tell you not only if it’s a chair or a table, which anyone can do,” Bala said, “but we can also tell you exactly which chair or table it is – an Ikea Mammut table or an Eames chair – as well as what style and material they appear to be.”

The tool is powering Facebook Marketplace today and identifies items by predicting color, style and material attributes, and by matching uploaded photos to clean catalog images. It uses a combination of deep learning (a kind of artificial intelligence) and a very large database of images.

“It can be tricky to market a product or dedicate time to creating listings,” Bell said. “GrokNet does the hard work for you.”

GrokNet is twice as accurate as Facebook’s prior systems at recognizing products, the company said, and improved coverage for its Home and Garden category from 33% to 90%.

“The system was trained on a diverse set of data, so it also works well for items that may look different depending on what part of the world you’re in,” Bell said.

Bell and Bala first reported their findings in a paper at the 2015 SIGGRAPH conference of the Association for Computing Machinery Special Interest Group on Computer Graphics and Interactive Techniques. They launched GrokStyle two years later, initially basing it at the Rev: Ithaca Startup Works business incubator. They soon raised $2 million in funding, partly through the Red Bear Angels, which connects investors with Cornell-led startups, and landed Ikea as a client.

“We provided visual recognition that was used in Ikea’s augmented reality app,” Bala said. “Augmented reality is becoming very exciting now; it has a lot of potential. In the case of furniture and home decor, augmented reality is particularly useful because people don’t have a good sense of size. With augmented reality, you can hold up your phone and overlay a piece of furniture, to see how it would fit in a corner of your room next to some other piece of furniture that you have.”

Bala was recently awarded ACM SIGRRAPH’s 2020 Computer Graphics Achievement Award for “fundamental contributions to physically based and scalable rendering, material modeling, perception for graphics and visual recognition,” including the GrokStyle research.

In January 2019, Facebook acquired GrokStyle, seeking technology in areas including visual recognition as part of its efforts to build e-commerce.

“So our team went over to Facebook, and within just one year they used GrokStyle’s technology as their starting point, rebuilt it and scaled for their global user base,” Bala said. “We’re really excited to see the impact it can have when it’s rolled out at the scale of Facebook’s user base and everybody’s using it around the world.”