Robert Barker/University Photography

The Merlin Bird Photo ID mobile app can identify hundreds of North American species it “sees” in photos and can now go anywhere its user goes.

The Merlin Bird Photo ID mobile app has been launched and, thanks to machine-learning technology, can identify hundreds of North American species it “sees” in photos.

The app was developed by Cornell Tech and California Institute of Technology computer vision researchers in partnership with the Cornell Lab of Ornithology and bird enthusiasts. Because Merlin Bird Photo ID can be used on mobile devices, it can go anywhere bird-watchers go.

“When you open the Merlin Bird Photo ID app, you’re asked if you want to take a picture with your smartphone or pull in an image from your digital camera,” said Merlin project leader Jessie Barry of the Lab of Ornithology. “You zoom in on the bird, confirm the date and location, and Merlin will show you the top choices for a match from among the 650 North American species it knows.”

Cornell Tech and Caltech computer scientists trained Merlin to recognize birds by showing it nearly 1 million photos that were collected and annotated by birders and volunteers mobilized by the Lab of Ornithology. Merlin scans its photo database for possible matches. Then, like any good birder, the system considers species that would be found at that specific time of year and in that location using information from the eBird program, which collects an average of 7 million bird observation records each month from around the world.

“In building Merlin Bird Photo ID, we were especially concerned with the quality and the organization of the data,” said Serge Belongie, professor of computer science at Cornell Tech. With Pietro Perona, the Allen E. Puckett Professor of Electrical Engineering at Caltech, Belongie is the co-founder of Visipedia, the Google-funded umbrella project that is using advances in machine learning and computer vision to help classify objects in photographs.

“Ultimately we want to create an open platform that any community can use to make a visual classification tool for butterflies, frogs, plants or whatever they need,” Belongie said.

Added Perona: “This app is the culmination of seven years of our students’ hard work and is propelled by the tremendous progress that computer vision and machine learning scientists are making around the world. All of a sudden, our smartphones can really see! This was a distant dream when I was a graduate student, and now it’s finally happening.”

How good is Merlin Bird Photo ID? Accuracy is around 90 percent if the user’s photo is of good quality, but the odds of getting an accurate match go down if the photo is fuzzy or if the bird is obscured.

Despite the high-tech advances, humans are still an important part of the process. The system requires eBird data from bird-watchers along with experts who can label photos used to train Merlin. Just around the corner is a Merlin Bird Photo ID release in Spanish for birds in Mexico. Down the road, the Merlin team will produce versions for South America, Europe, Asia, Africa and Australia.

“The wonderful thing about this project is the collaboration with the Visipedia team,” Barry said. “We have a product that really works because it’s supported by fantastic research, and is great for the birding community because it’s built for birders by birders.”

Graduate student Grant Van Horn and postdoctoral researcher Steve Branson, both at Caltech, are part of the Visipedia team and developed the algorithms that allow Merlin to learn to recognize the birds.

The Merlin Bird Photo ID can be downloaded free for iOS or Android systems from the Apple and Google Play app stores. It is now included in the Merlin Bird ID app, originally released in 2014 to identify birds by asking users five questions about birds they saw.

Merlin Photo ID is powered by Visipedia with support from Google, the Jacobs Technion-Cornell Institute and the National Science Foundation.