6 min read

Analyzing the $35 Premium Plan Experiment

On October 16, 2019 we introduced a new Premium plan that costs $35 per month to a randomly selected group of users. We hypothesized that, by adding a cheaper Premium plan to the pricing page, we would see an overall increase in the number of customers that subsribed to a Premium plan and increase total MRR.

TLDR

As of November 4, 2019, users in the enabled group of that were presented with the $35/mo Premium plan have converted at higher rates and have generated significantly more MRR than users in the control group.

More people in the control group subscribed to the $65 Premium plans, however this difference is made up for by the number of customers that subscribed to the $35 Premium plan.

It’s worth noting that ARPU is lower for users in the enabled group, however that is more than offset by the presence of more customers that subscribed to Premium and Business plans.

Data Collection

To analyze the results of this experiment, we’ll collect all of the active subscriptions of users that were placed in one of the two experiment groups. We use the SQL query below to retrieve this data from BigQuery.

# connect to bigquery
con <- dbConnect(
  bigrquery::bigquery(),
  project = "buffer-data"
)

# define sql query
sql <- "
  select distinct
    e.id
    , e.original_timestamp as enrolled_at
    , e.experiment_group
    , e.user_id as account_id
    , a.publish_user_id as user_id
    , u.billing_plan
    , u.stripe_customer_id as customer_id
    , s.id as subscription_id
    , s.plan_id 
  from segment_publish_server.experiment_enrolled_view e
  left join dbt_buffer.core_accounts a
    on e.user_id = a.id
  left join dbt_buffer.publish_users u
    on a.publish_user_id = u.id
  left join dbt_buffer.stripe_paid_subscriptions s
    on s.customer_id = u.stripe_customer_id
    and s.status = 'active'
    and s.plan_id not like '%analyze%'
  where e.experiment_id = 'eid20_premium_1user_35dollars_payday'
"
  
# query BQ
users <- dbGetQuery(con, sql)

There are around 38K users that have been enrolled in this experiment.

Exploratory Analysis

First let’s look at how many users are on each plan. The list below displays users current plans. There are a couple interesting things to note:

  • There are a bunch of Pro users in these groups. That’s because we enrolled Pro customers and Pro trialists in the experiment, which means that their subscriptions could have begun before we started the experiment. We’ll end up excluding them from the analysis.

  • There’s one customer on the $35/mo Premium plan that got enrolled in the control group. After investigating, I found out that this user is Char. We’ll remove her user from the dataset.

  • There are users on Free and legacy Awesome plans that were enrolled because they had started Pro trials.

plan_id control enabled
pro_v1_monthly 5805 5654
pro_v1_yearly 4123 4294
publish_premium_m_35_201910 1 34
premium_business_v1_monthly 26 15
pro15_v1_monthly 16 13
business_v2_small_monthly 7 11
pro15_v1_yearly 6 9
publish_premium_y_336_201910 NA 9
premium_business_v1_yearly 12 7
business_v2_small_yearly 3 2
pro-monthly 1 1
pro-annual 4 NA

Effect on Total Revenue

Now let’s assign MRR amounts to each plan id and estimate total MRR for each group. The code for assigning MRR amounts to plans has been hidden.

experiment_group users subs total_mrr arpu
control 16997 10005 $141,627 $8.33
enabled 16945 10051 $142,235 $8.39

This table includes all users enrolled in the experiment, including Pro users. We can see that total MRR and ARPU are slightly higher for users in the enabled group.

Effect on Premium and Business Revenue

Because users could have been on the Pro plan before being enrolled, it might be more useful to only look at users on Premium and Business plans. We’ll do that in the table below. The code has been hidden again.

experiment_group customers subs total_mrr arpu
control 48 49 $3,527.48 $73.49
enabled 78 79 $4,160.07 $53.33

As you can see, Premium and Business MRR is significantly higher for users in the enabled group. This validates the hypothesis that offering a lower priced Premium plan would result in more customers subscribing to Premium.

It’s worth noting that ARPU is lower for users in the enabled group, however that is more than offset by the presence of more customers that subscribed to Premium and Business plans.

Effect on Premium

Let’s narrow our focus and look only at the Premium plans. We can see that more people in the control group subscribed to the $65 Premium plans, however this difference is compensated for by the number of customers that subscribed to the $35 Premium plan.

Total Premium MRR is higher in the enabled group ($2804 compared to $2483), even though ARPU is lower ($43.13 compared to $65.34).

plan_id experiment_group customers subs total_mrr arpu
premium_business_v1_monthly control 26 26 $1,820 $70
premium_business_v1_monthly enabled 15 15 $975 $65
premium_business_v1_yearly control 12 12 $663.00 $55.25
premium_business_v1_yearly enabled 7 7 $386.75 $55.25
publish_premium_m_35_201910 enabled 34 34 $1,190 $35
publish_premium_y_336_201910 enabled 9 9 $252 $28

Subscription Started and Upgraded Events

Let’s gather these events from Mixpanel. These are all of the subscription events in Mixpanel that were triggered by users assigned to an experiment group.

library(RMixpanel)

# set up project
account = mixpanelCreateAccount("Buffer",
                                token = Sys.getenv("MIXPANEL_TOKEN"),
                                secret = Sys.getenv("MIXPANEL_SECRET"), 
                                key = Sys.getenv("MIXPANEL_KEY"))

# get enrollment events
enrollment_events <- mixpanelGetEvents(account,
                            where = 'properties["experimentId"] == "eid20_premium_1user_35dollars_payday"',
                            from = "2019-10-17",
                            df = TRUE,
                            verbose = FALSE)

# get subscription events
subscription_events <- mixpanelGetEvents(account,
                            event = list("Subscription Started",
                                         "Subscription Upgraded",
                                         "Subscription Downgraded"),
                            from = "2019-10-18",
                            to = "2019-11-01",
                            df = TRUE,
                            verbose = FALSE)

# join events
events <- subscription_events %>% 
  inner_join(select(enrollment_events, distinct_id, experimentGroup), by = "distinct_id") %>% 
  select(event, email, distinct_id, event, experimentGroup, planId, previousPlanId, product) %>% 
  filter(product != "analyze")

Let’s look at the types of events that were returned.

# count event types 
events %>% count(event, sort = TRUE)
## # A tibble: 3 x 2
##   event                       n
##   <chr>                   <int>
## 1 Subscription Started       83
## 2 Subscription Upgraded      31
## 3 Subscription Downgraded    15

Wow, it looks like there were 15 downgrade events to the new premium plan. Let’s look at the plans that users downgraded from.

# look at downgrade events
events %>% 
  filter(event == "Subscription Downgraded") %>% 
  count(previousPlanId, sort = TRUE)
## # A tibble: 4 x 2
##   previousPlanId                  n
##   <chr>                       <int>
## 1 pro_v1_yearly                  10
## 2 premium_business_v1_monthly     2
## 3 premium_business_v1_yearly      2
## 4 publish_premium_m_35_201910     1

Two users downgraded from the yearly Premium 35 plan, and ten downgraded from the Pro Yearly plan. Those should really be upgrades eh. Let’s change the type of event for those events.

# switch downgraded events to upgraded
events <- events %>% 
  mutate(event = ifelse(event == "Subscription Downgraded" & 
                          (planId == "publish_premium_m_35_201910" | 
                             planId == "publish_premium_y_336_201910") &
                          previousPlanId == "pro_v1_yearly", "Subscription Upgraded", event))

Now, let’s look at all the subscription started and upgraded events for users in each group.

## # A tibble: 9 x 3
##   planId                       control enabled
##   <chr>                          <int>   <int>
## 1 pro_v1_monthly                    32      31
## 2 pro_v1_yearly                      8      11
## 3 publish_premium_m_35_201910       NA       5
## 4 premium_business_v1_monthly        4       3
## 5 business_v2_small_monthly          2       2
## 6 premium_business_v1_yearly         4       2
## 7 publish_premium_y_336_201910      NA       2
## 8 pro15_v1_monthly                  NA       1
## 9 business_v2_small_yearly           1      NA

Looks like we may be missing a couple Mixpanel events here, but still useful to see. :)