What Do Your App’s User Retention Numbers Really Mean?

Is your app sticky? Making sense of those user retention numbers is all about context, according to a new report.

What Do Your App’s User Retention Numbers Really Mean?
[Image: Flickr user With Associates]

For app developers, the goal is to always be growing and reaching new users, but those users aren’t as valuable if they don’t stick with the app. These user retention metrics are often poorly understood, according to new data from Mixpanel, a San Francisco-based analytics firm. With a little bit of context, developers could better understand how apps stack up to each other–and gain a big competitive edge in the process.


Mixpanel wanted to look at retention rates in different app categories like messaging, social, e-commerce, and education, so they pooled data from clients and began running the numbers on how many users opened an app more than once a month over a three-month period. Then they broke down what average retention was in each category. The range was big. Messaging apps had more than a 50% retention rate compared to music apps, for example, which topped out around 15%. The goal was to pull out subtle variations rather than over-simplifying diverse data.

In July 2011, Fred Wilson, cofounder of Union Square Ventures, blogged about user retention in apps and laid out a ratio for retention the he said emerged over and over in the analytics data Union Square had access to for different companies. He wrote:

I call this ratio 30/10/10 and so many services that we see exhibit it within a few percentage points . . . 30% of the registered users or number of downloads will use the service each month, 10% of the registered users or number of downloads will use the service each day, the max number of concurrent users of a real-time service will be 10% of the number of daily users.

Redefining Retention: Context Matters

Wilson’s 30/10/10 formula became something of an industry standard for thinking about retention. Thirty percent of people who had the app should use it at least once a month after the initial download. Sounds reasonable. But Nicole Leverich, the director of corporate communications at Mixpanel, says that that number may not be right for everyone, and in fact is probably wrong for apps in almost every genre, simply because it’s too specific to generalize to such a varied industry.

“Retention is a big thing that everybody wants to track,” Leverich says. “But while people are using Mixpanel to measure the retention in their app, they don’t have context for whether it’s good or not. You can’t assume one number across very different types of applications. Our goal was to try to give companies more specific benchmarks by vertical, so they can measure themselves against a standard, and have a decent guess of how they’re doing for their space. There’s a huge variation, which I don’t think is a shock.”

The report looks at data from August, September, and October 2013. Leverich wanted to compare across multiple months to smooth out incidental variation and get an average of how many users log in to different apps at least twice per month. Then she divided the data into genre categories so comparable apps could be measured together with fewer false comparisons. “Retention can get to be a really complex topic, and that’s probably why there’s not a lot of benchmark data on it,” Leverich says.

She points out that a messaging app might look at retention on a daily basis, because the goal is to get users hooked on the service. But for something like a baby supply app, users might only open the app once every six weeks to order another case of diapers. Just because retention numbers would seem extremely low for the baby supply app wouldn’t necessarily mean that the app wasn’t popular.


When Average Isn’t Good Enough

So what happens when an app hits a retention rate on par with its genre’s average? Mission accomplished? Hardly. Leverich says that apps need to continue growing their retention rather than simply treading water once they hit the industry average (as calculated by Mixpanel or through any data a developer has access to). The takeaway from the report, she says, shouldn’t be that having average retention is all an app needs. And Leverich says that retention rate may be a somewhat underexplored area of growth since companies aren’t always looking at tailored retention averages by category when they compare their own rates.

One of Leverich’s more surprising findings from the Mixpanel data is that “instant gratification” apps (ones that users can begin interacting with instantly without needing to sign in or identify themselves) actually have a slightly lower average retention rate across all categories combined than “customization” apps where the user has an initial barrier, however large or small, to using the app in a meaningful way. “The fact that the customization apps had the higher rate of retention, it wasn’t what I expected,” Leverich says. “But thinking about it a little bit more, once you’ve entered information to customize an app, you’re more incentivized to go back.”

Though the comparison between “instant gratification” apps and “customization” apps raises interesting questions about how those two categories of app perform in the marketplace, the data was generalized across all app genres, and the retention rate difference between the two groups was only 6%.

Leverich acknowledges this and is also quick to point out that the data in the report overall is only a starting point and may be unintentionally weighted by the swath of client data Mixpanel happens to have access to. “I try to be very up front with the fact that this is based on the thousands of apps that use Mixpanel, so that means that there’s self-selection. But I do still think it’s fairly representative. And honestly, if this results in a bit of a debate about what retention is or should be by vertical, that’s only a positive thing. More data and more insights are really valuable in the appdev community.”

Certainly the community would benefit from more raw data or analysis of retention rates. And for retention to really become a more visible metric it needs to be mapped consistently over time and in all quarters of the year. Mixpanel’s report is one step, though, toward creating benchmarks for retention and evaluating how sticky apps really are.