When you marvel at the brilliant colors of a peacock’s tail or enjoy the trill of a songbird, you might pause to consider that those creatures’ ancestors probably didn’t have a lot of friends.

Many animals display visual or auditory cues that contain messages like, “I would be a desirable mate” or “I am very strong; don’t come into my territory.” Species that lack such “quality signals” – including humans, crows and dolphins – probably evolved in smaller social groups where everyone already knows what everyone else is like, according to Michael Sheehan, assistant professor of neurobiology and behavior.

Sheehan and Thore J. Bergman, associate professor of ecology and evolutionary biology at the University of Michigan, set forth this idea in an invited review that is the cover story in the January – February 2016 issue of the journal Behavioral Ecology.

“Judging rivals by their ornaments is sort of like judging a book by its cover. You’ll get some information about the book, but you’ll learn much more by reading it,” Sheehan said. “Animals that repeatedly interact with the same individuals essentially get a chance to ‘read the book’ and learn about their rivals.”

As species evolved, he said, quality signals and social recognition have been antagonistic, with selection depending on the social structure in which the animals lived. Evolving in small social groups favors social recognition over external signals.

In a troop of baboons, everyone knows who the alpha male is. But Gelada monkeys that live in huge herds in the highlands of Ethiopia carry a bright red chest patch that indicates status. The Gelada chest patch is thought to act like a lion’s mane, advertising the strength of a male to unfamiliar rivals that might challenge him for his pride.

Each form of signaling requires an investment: It takes energy to grow a bigger mane or develop a more flexible vocal system, but social recognition requires more learning, which is energetically costly too. Experiments with fruit flies have shown that flies that had to learn a task died quicker when later placed under stress. The benefits of the signaling system have to outweigh the costs both for senders and receivers as traits co-evolve.

“What this broadly tells us is why some animals are so brilliantly decorated and have invested a lot in display,” Sheehan said, “and why social species lack these designs. We might notice [physical] things about a species that would give some sense of their social system, and if you know their social system you could predict their signaling.”

It’s not an all-or-nothing game, the co-authors point out. For example, when a songbird establishes its territory, quality signaling plays a major role. But birds typically learn about their territorial neighbors and then depend on individual recognition instead of quality signals. In the case of the Gelada’s chest patch and lion’s mane, the message is likely meant just for unfamiliar rival males.

“Humans are an interesting case,” Sheehan noted. “Cultural adaptation may have outpaced biological adaptation.” We evolved to use social interaction, but we have incorporated quality signals into the culture, from military uniforms to the number of windows in an office.

This way of thinking opens new ideas to explore, the co-authors said. The development of quality signals might predispose a species to move away from living in small social groups, and vice versa.

The journal article will be followed by extensive commentary and discussion, Sheehan said. “It’s a way of thinking about this but still largely untested,” he said.

The various authors have been supported by the National Institutes of Health, the National Science Foundation and the University of Michigan.