On October 16, 2019 we introduced a new Premium plan that costs $35 per month to a randomly selected group of users. We hypothesized that, by adding a cheaper Premium plan to the pricing page, we would see an overall increase in the number of customers that subsribed to a Premium plan and increase total MRR.
As of November 4, 2019, users in the
enabled group of that were presented with the $35/mo Premium plan have converted at higher rates and have generated significantly more MRR than users in the
More people in the control group subscribed to the $65 Premium plans, however this difference is made up for by the number of customers that subscribed to the $35 Premium plan.
It’s worth noting that ARPU is lower for users in the
enabled group, however that is more than offset by the presence of more customers that subscribed to Premium and Business plans.
To analyze the results of this experiment, we’ll collect all of the active subscriptions of users that were placed in one of the two experiment groups. We use the SQL query below to retrieve this data from BigQuery.
# connect to bigquery con <- dbConnect( bigrquery::bigquery(), project = "buffer-data" ) # define sql query sql <- " select distinct e.id , e.original_timestamp as enrolled_at , e.experiment_group , e.user_id as account_id , a.publish_user_id as user_id , u.billing_plan , u.stripe_customer_id as customer_id , s.id as subscription_id , s.plan_id from segment_publish_server.experiment_enrolled_view e left join dbt_buffer.core_accounts a on e.user_id = a.id left join dbt_buffer.publish_users u on a.publish_user_id = u.id left join dbt_buffer.stripe_paid_subscriptions s on s.customer_id = u.stripe_customer_id and s.status = 'active' and s.plan_id not like '%analyze%' where e.experiment_id = 'eid20_premium_1user_35dollars_payday' " # query BQ users <- dbGetQuery(con, sql)
There are around 38K users that have been enrolled in this experiment.
First let’s look at how many users are on each plan. The list below displays users current plans. There are a couple interesting things to note:
There are a bunch of Pro users in these groups. That’s because we enrolled Pro customers and Pro trialists in the experiment, which means that their subscriptions could have begun before we started the experiment. We’ll end up excluding them from the analysis.
There’s one customer on the $35/mo Premium plan that got enrolled in the control group. After investigating, I found out that this user is Char. We’ll remove her user from the dataset.
There are users on Free and legacy Awesome plans that were enrolled because they had started Pro trials.
Effect on Total Revenue
Now let’s assign MRR amounts to each plan id and estimate total MRR for each group. The code for assigning MRR amounts to plans has been hidden.
This table includes all users enrolled in the experiment, including Pro users. We can see that total MRR and ARPU are slightly higher for users in the enabled group.
Subscription Started and Upgraded Events
Let’s gather these events from Mixpanel. These are all of the subscription events in Mixpanel that were triggered by users assigned to an experiment group.
library(RMixpanel) # set up project account = mixpanelCreateAccount("Buffer", token = Sys.getenv("MIXPANEL_TOKEN"), secret = Sys.getenv("MIXPANEL_SECRET"), key = Sys.getenv("MIXPANEL_KEY")) # get enrollment events enrollment_events <- mixpanelGetEvents(account, where = 'properties["experimentId"] == "eid20_premium_1user_35dollars_payday"', from = "2019-10-17", df = TRUE, verbose = FALSE) # get subscription events subscription_events <- mixpanelGetEvents(account, event = list("Subscription Started", "Subscription Upgraded", "Subscription Downgraded"), from = "2019-10-18", to = "2019-11-01", df = TRUE, verbose = FALSE) # join events events <- subscription_events %>% inner_join(select(enrollment_events, distinct_id, experimentGroup), by = "distinct_id") %>% select(event, email, distinct_id, event, experimentGroup, planId, previousPlanId, product) %>% filter(product != "analyze")
Let’s look at the types of events that were returned.
# count event types events %>% count(event, sort = TRUE)
## # A tibble: 3 x 2 ## event n ## <chr> <int> ## 1 Subscription Started 83 ## 2 Subscription Upgraded 31 ## 3 Subscription Downgraded 15
Wow, it looks like there were 15 downgrade events to the new premium plan. Let’s look at the plans that users downgraded from.
# look at downgrade events events %>% filter(event == "Subscription Downgraded") %>% count(previousPlanId, sort = TRUE)
## # A tibble: 4 x 2 ## previousPlanId n ## <chr> <int> ## 1 pro_v1_yearly 10 ## 2 premium_business_v1_monthly 2 ## 3 premium_business_v1_yearly 2 ## 4 publish_premium_m_35_201910 1
Two users downgraded from the yearly Premium 35 plan, and ten downgraded from the Pro Yearly plan. Those should really be upgrades eh. Let’s change the type of event for those events.
# switch downgraded events to upgraded events <- events %>% mutate(event = ifelse(event == "Subscription Downgraded" & (planId == "publish_premium_m_35_201910" | planId == "publish_premium_y_336_201910") & previousPlanId == "pro_v1_yearly", "Subscription Upgraded", event))
Now, let’s look at all the subscription started and upgraded events for users in each group.
## # A tibble: 9 x 3 ## planId control enabled ## <chr> <int> <int> ## 1 pro_v1_monthly 32 31 ## 2 pro_v1_yearly 8 11 ## 3 publish_premium_m_35_201910 NA 5 ## 4 premium_business_v1_monthly 4 3 ## 5 business_v2_small_monthly 2 2 ## 6 premium_business_v1_yearly 4 2 ## 7 publish_premium_y_336_201910 NA 2 ## 8 pro15_v1_monthly NA 1 ## 9 business_v2_small_yearly 1 NA
Looks like we may be missing a couple Mixpanel events here, but still useful to see. :)