People love rankings – but do they really mean all that much?

Sometimes they do, depending on several factors, according to Wyatt Lee, assistant professor in the Nolan School of Hotel Administration, in the SC Johnson College of Business.

Specifically, Lee found, a drop in rank is more impactful than a rise of similar magnitude. The availability of other information, a high prior position and the sophistication of the audience can also lessen the effect of a ranking.

“Rankings are everywhere today. There’s research that says rankings are important, but at the same time, other studies show they’re actually not so important,” said Lee, author of “Falling Fortunes: The Contingent and Asymmetric Effect of Rankings on Organizational Outcomes,” published Sept. 3 in Administrative Science Quarterly.

“Some researchers suggest people don’t really use them, or that they don’t really have an effect on our decisions,” Lee said. “So this research started with the motivation of trying to understand, OK, when do they really matter?”

For this study, Lee first compared how vacillations in a firm’s Fortune 500 rank were associated with its stock returns. He controlled for revenue changes to address the concern that investors might be responding to revenue information as opposed to rankings.

“One of the difficult things when studying rankings is that they’re correlated with so many things, such as performance” Lee said. “The Fortune 500 is a ranking of revenue, so what’s really hard to know is, are people reacting to the rankings, or are they reacting to the fall in revenue, or to the event that caused this fall?”

He found that a drop in ranking had a measurable impact on a firm’s stock price, while a similar rise had no discernible effect.

Lee then exploited a natural experiment condition: In 1994, the Fortune 500 broadened its industry coverage to include firms in the service sector. This change allowed more than 200 service firms – including Walmart, AT&T and Sears Roebuck, which all cracked the top 10 – to enter the rankings.

Due to the influx of service firms, many non-service firms – primarily from manufacturing and energy production – ranked in the Fortune 500 list experienced a significant decline in the ’94 rankings. That same year, Fortune introduced its Fortune 1000, which gave Lee the ability to precisely measure the decline of firms that dropped to between 501 and 1,000.

Lee found a similar but less pronounced effect in the experiment condition, likely showing that people respond to a drop in rankings, even when it is not driven by a substantial drop in performance or revenue.

Lee also found that the Fortune 500 rankings mattered more for organizations ranked lower on the list, as well as those for which there is less alternative information, such as media coverage.

Audience sophistication can also be particularly telling in terms of whether rankings are important, Lee said.

“Prior research often treats individuals as a homogenous group, but people use rankings differently,” he said. “We often think that people just look at the rankings, but they also read newspapers, they look at analyst reports, and they compare and use this information as complements to rankings.”

One big takeaway for Lee: the asymmetry in rankings movement, where a drop is more impactful than a rise in most cases.

“It could have to do with what we call ‘reputation anxiety,’” he said. “The fear of falling often drives firms to take certain actions to avoid a bigger drop in the future.”