When a person needs to fix an appliance or a car, they might dig out the manual, watch a few videos online and make a plan before diving in. What if a robot could do the same thing?

“That’s essentially the aspirational goal,” said Sanjiban Choudhury, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science, who leads the People and Robot Teaching and Learning (PoRTaL) group. “We want robots to look at human videos and be able to do the task themselves.”

Choudhury just received a three-year, $750,000 Young Investigator Program award from the Office of Naval Research (ONR) to develop new ways to train robots to perform complex, multistep tasks, such as inspecting and repairing ship engines. He aims to provide proof of concept that robots can integrate information from repair manuals, videos of people doing the task and real-time human feedback to perform complicated jobs.

This advance would allow autonomous robots to work in hazardous environments, like cargo holds and engine rooms, or even repair broken items around the house.

Currently, robots are very good at doing precision tasks, like fetching a hammer or turning a screw, but struggle to plan and perform more complex operations. Likewise, artificial intelligence (AI)-powered large language models, similar to ChatGPT, successfully parse text in manuals and interpret videos of humans, but can’t yet communicate with robots. The goal of this project is to bridge these technologies so that robots can accomplish more complex tasks with the help of AI, and even course-correct when things don’t go as planned.

A major challenge will be to translate written and spoken language and human actions into a series of robot movements – an effort called “common grounding.”

“The way humans do a task is very different from the way robots do tasks,” Choudhury said. For example, humans can move more fluidly and use two hands, whereas robots might have just one arm and have more limited movements. “We need to establish some kind of alignment between what actions the human does versus the actions the robot does.”

Choudhury is one of 24 recipients of the 2025 ONR Young Investigator Program award, which drew from more than 230 applicants. The program awards outstanding early-career academics in STEM fields and supports them in developing innovative solutions to challenges impacting the Navy and Marine Corps. Choudhury’s award falls under the Human Interaction with Autonomous Systems program, an initiative to create intelligent autonomous systems and robots that can act as teammates alongside humans.

The award will fund two graduate students, as well as the equipment and computer parts needed to train and test robots already in the PoRTaL lab. The team works with a robotic arm that can manipulate tools, such as hammers and screwdrivers, and a mobile manipulator robot that can fetch and deliver objects. Their approach and the resulting training models will all be open source so that other robotics researchers can apply their methods.

“It’s a pretty big shift from what robots can do today,” Choudhury said, noting that his group has had some success with translating human videos to robot tasks in preliminary work, and he looks forward to building on that foundation. 

“I’m excited about taking a completely new task and just handing that to the system and seeing what it can do,” he said. “I don’t think any research group has pushed the limits of how general-purpose robots can be.”

Patricia Waldron is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.