More than 40% of U.S. adults have experienced some form of online harassment, according to Pew Research surveys, highlighting the need for content moderation on social media, which helps prevent and remove offensive or threatening messages.

But who – or what – are the moderators policing the cyber landscape? And can they be trusted to act as gatekeepers for safe content?

An interdisciplinary collaboration with Marie Ozanne, assistant professor of food and beverage management at the Peter and Stephanie Nolan School of Hotel Administration, in the Cornell SC Johnson College of Business, found that both the type of moderator – human or AI – and the “temperature” of the harassing content influenced people’s perception of the moderation decision and the moderation system.

Ozanne is first author of “Shall AI Moderators Be Made Visible? Perception of Accountability and Trust in Moderation Systems on Social Media Platforms,” which published in August in Big Data & Society. Members of Cornell Social Media Lab, from the Department of Communication in the College of Agriculture and Life Sciences, contributed to this research.

The study used a custom social media site, on which people can post pictures of food and comment on other posts. The site contains a simulation engine, Truman, an open-source platform that mimics other users’ behaviors (likes, comments, posts) through preprogrammed bots created and curated by researchers.

The Truman platform – named after the 1998 film, “The Truman Show” – was developed at the Cornell Social Media Lab, which is led by Natalie Bazarova, professor of communication (CALS). Co-author Dominic DiFranzo, a former postdoctoral researcher in the Social Media Lab and now an assistant professor of computer science and engineering at Lehigh University, was a lead developer of Truman.

“The Truman platform allows researchers to create a controlled yet realistic social media experience for participants, with social and design versatility to examine a variety of research questions about human behaviors in social media,” Bazarova said. “Truman has been an incredibly useful tool, both for my group and other researchers to develop, implement and test designs and dynamic interventions, while allowing for the collection and observation of people’s behaviors on the site.”

For the study, Ozanne and Bazarova’s group recruited nearly 400 participants and told them they’d be beta testing a new social media platform. They were asked to log in at least twice a day for two days, and were randomly assigned to one of six experiment conditions, varying both the type of content moderation system (others users; AI; no source identified) and the type of harassment comment they saw (ambiguous or clear).

The researchers found that users are generally more likely to question AI moderators, especially how much they can trust their moderation decision and the moderation system’s accountability, but only when content is inherently ambiguous. For a more clearly harassing comment, trust in AI, human or an unknown source of moderation was approximately the same.

“It’s interesting to see,” Ozanne said, “that any kind of contextual ambiguity resurfaces inherent biases regarding potential machine errors.”

Ozanne said trust in the moderation decision and perception of system accountability – i.e., whether the system is perceived to act in the best interest of all users – are both subjective judgments, and “when there is doubt, an AI seems to be questioned more than a human or an unknown moderation source.”

The researchers suggest that future work should look at how social media users would react if they saw humans and AI moderators working together, with machines able to handle large amounts of data and humans able to parse comments and detect subtleties in language.

“Even if AI could effectively moderate content,” they wrote, “there is a [need for] human moderators as rules in community are constantly changing, and cultural contexts differ.”

Aparajita Bhandari, a doctoral student in the field of communication and a member of the Social Media Lab, was second author. Bhandari was lead author of a paper in 2021, from the same research team, on bystander behaviors and content moderation practices, which showed that providing visibility around the source of moderation may hinder users’ likelihood to engage in prosocial behaviors.

The current research was supported by a grant from the U.S. Department of Agriculture’s National Institute of Food and Agriculture, through its Hatch Project program.