Google Analytics just incorporated a new feature that makes Cohort Analysis easier then ever. This might be one of the coolest changes to Analytics in the past year. If you have not heard about Cohort Analysis before this is a quick review.
We are constantly making changes to optimize our website, but measuring the true impact of those changes on different visitor segments over time is difficult with traditional testing tools. Cohort Analysis is a powerful tool for optimization that allows you to track how your audience or a specific demographic engages with your website as well as different marketing tactics through time. Standard analytic packages already allow us to attribute actions to specific audiences based on variables such as; demographics, device, geography. Most of incorporate this data into the A/B and/or multivariate testing that we do but we seldom think of the limitations of these common practices. The biggest issue with A/B and multivariate testing methods is that they only give a snapshot of performance during a specific time interval.
Cohort analysis allows us to take things a step further and look at how specific audiences interact with our website through time. Let’s say in January you decide to offer a subscription service that is normally $39.95 per a month for $19.95 with a plan to increase back to the normal rate in February. The question is what happens to these $19.95 subscribers overtime? Do they have a higher retention rate then the subscribers being charged more? Do they refer more friends or use your service differently? Maybe they are not as engaged since they are paying less and they correlate how much they pay with the value of the service. These are the perfect types of questions to answer with Cohort Analysis.
A cohort is defined as “a group of people who share a common characteristic or experience within a defined period.” Here is a simple example looking at users that signed up in different months and retention rates for each group for the following 3 months. Looking at this example one might guess that some changes were made on the site between month 1 and 3 that account for marginal improvement in retention in those members who joined in the 3rd month.
How much do these numbers mean? Actually, without knowing the sample size it’s difficult to judge if this improvement is significant ( what is significance? ).
Medical studies and drug trails have been making use of Cohort Analysis for decades but when it comes to websites or mobile a cohort can be defined as any group of users that completed a certain action (such as install, uploaded photo, made comment) with a given time frame.