Study: Inflation of Online Ratings Can Be Beneficial and Detrimental
Many adults consult online ratings and reviews before they make a purchase. However, how do ratings affect sales is a challenging topic as evidenced from controversy surrounding Rotten Tomato scores or sellers trying to manipulate ratings on various platforms. A new study investigates how rating inflation affected a digital platform as well as the choices users made. The study found that inflation can have benefits and detriments: While sales rose, users’ trials declined, and sales were concentrated among popular sellers. Rating inflating is when platforms change their strategy to increase average ratings. So, users see higher ratings for all restaurants.
The study, by researchers at Carnegie Mellon University (CMU) and the University of British Columbia, appears in Information Systems Research.
“Overall, our results illustrate the potential consequences of rating inflation that platforms need to consider when designing and managing their rating systems,” says Rahul Telang, professor of information systems and management at CMU’s Heinz College, who coauthored the study.
In the process of rating inflation, the variance across restaurants go down – since restaurants tend to have higher ratings, they look more similar. For instance, on eBay, the median seller might have a 100 percent positive rating, while the seller in the bottom 10th percentile has a 98 percent positive rating. Not only are these exceptionally high positive ratings, but there is little variation between good and mediocre sellers, making the overall ratings less informative for consumers.
In this study, researchers conducted an experiment with a digital food-delivery platform in a large Asian city that changed its rating in a neighborhood (with 48 restaurants on the platform), which resulted in rating inflation. They examined the impact of this change over 13 weeks in mid-2017 on users’ purchases, trials of new restaurants, and sales concentration in the context of choosing restaurants on the platform.
The study found that the platform benefitted from one aspect of rating inflation: Because this type of inflation makes sellers’ products appear to be of higher quality, both users’ purchases and sellers’ sales increased in the short term due to increased average ratings.
However, there were also negative consequences: The decrease in rating variance reduced the informativeness of ratings, which increased the uncertainty of users’ perceptions of restaurants’ quality and made them less likely to try a new establishment. Since perceptions of ratings were less certain, they became less important in shaping users’ perceptions of quality, and this boosted the importance of other signals (e.g., prior beliefs, prior experience), leading to a greater concentration of sales among popular restaurants.
Among the study’s limitations, the authors note they did not investigate the long-term effects of rating inflation. In addition, their data are from one city, limiting the generalizability of their results.
“Our findings offer important insights about the tradeoffs of rating inflation for managers, designers, and developers of digital platforms that use ratings to help users choose among numerous sellers,” suggests Hui Li, associate professor of marketing at CMU’s Tepper School of Business, who coauthored the study. “Such individuals should build informativeness into the design of rating systems to account for and minimize rating inflation on their platforms.”
###
Summarized from an article in Information Systems Research, The Consequences of Rating Inflation on Platforms: Evidence from a Quasi-Experiment by Aziz, A (University of British Columbia), Li, H (Carnegie Mellon University), and Telang, R (Carnegie Mellon University). Copyright 2022. All rights reserved.
About Heinz College of Information Systems and Public Policy
The Heinz College of Information Systems and Public Policy is home to two internationally recognized graduate-level institutions at Carnegie Mellon University: the School of Information Systems and Management and the School of Public Policy and Management. This unique colocation combined with its expertise in analytics set Heinz College apart in the areas of cybersecurity, health care, the future of work, smart cities, and arts & entertainment. In 2016, INFORMS named Heinz College the #1 academic program for Analytics Education. For more information, please visit www.heinz.cmu.edu.