Activision Study Indicates Lower Player Retention Without Skill-Based Matchmaking

A recent study by Activision suggests that players are less likely to continue playing online multiplayer games without skill-based matchmaking. Activision released a 25-page document featuring their research, published online and noted by GamesIndustry.biz, which involved an unannounced experiment with Call of Duty: Modern Warfare III in early 2024.

Typically, Modern Warfare III uses a player’s past performance, such as kills and deaths, to pair them with similarly skilled opponents, creating a balanced and competitive experience. For this experiment, Activision targeted the North American player base and divided them into two groups. One group experienced a ‘Deprioritise Skill Test’, where the matchmaking algorithm placed less emphasis on skill, while the other group continued with the usual skill-based matchmaking system.

Findings revealed that players subjected to the ‘Deprioritise Skill Test’ were less inclined to return to the game. Although the top 10% of skilled players revisited the game more frequently within two weeks (due to winning more matches), the remaining 90% were less likely to come back.

Activision highlights a significant concern for top-tier players: a shrinking player base could result in higher-ranked players moving down the ranks due to fewer competitors. “Ultimately, this will result in a worse experience for all players, as there will be fewer and fewer players available to play with,” the company stated.

The study also noted that the experiment only reduced the emphasis on skill in matchmaking rather than eliminating it entirely.

“If it were completely removed, we would expect to see the player population erode rapidly in the span of a few months, resulting in a negative outcome for all our players.”

Additionally, Activision conducted experiments where skill-based matchmaking was even more stringent, which resulted in the opposite effect: top players were more likely to drop out, while the bottom 80% were more likely to return.

“This change was not rolled out as a standard approach, as we continue to strive for a balance in our approach to matchmaking,” Activision concluded.

While I personally disagree with the findings, I do appreciate that Activision is trying to listen to their player base and even went to the extent of doing an experiment. This study underscores the critical role skill-based matchmaking plays in maintaining a healthy and engaged player base in Call of Duty.