Analyzing the $35 Premium Plan Experiment

TLDR

As of November 4, 2019, users in the enabled group of that were presented with the $35/mo Premium plan have converted at higher rates and have generated significantly more MRR than users in the control group.

More people in the control group subscribed to the $65 Premium plans, however this difference is made up for by the number of customers that subscribed to the $35 Premium plan.

It’s worth noting that ARPU is lower for users in the enabled group, however that is more than offset by the presence of more customers that subscribed to Premium and Business plans.

Data Collection

To analyze the results of this experiment, we’ll collect all of the active subscriptions of users that were placed in one of the two experiment groups. We use the SQL query below to retrieve this data from BigQuery.

# connect to bigquery
con <- dbConnect(
  bigrquery::bigquery(),
  project = "buffer-data"
)

# define sql query
sql <- "
  select distinct
    e.id
    , e.original_timestamp as enrolled_at
    , e.experiment_group
    , e.user_id as account_id
    , a.publish_user_id as user_id
    , u.billing_plan
    , u.stripe_customer_id as customer_id
    , s.id as subscription_id
    , s.plan_id 
  from segment_publish_server.experiment_enrolled_view e
  left join dbt_buffer.core_accounts a
    on e.user_id = a.id
  left join dbt_buffer.publish_users u
    on a.publish_user_id = u.id
  left join dbt_buffer.stripe_paid_subscriptions s
    on s.customer_id = u.stripe_customer_id
    and s.status = 'active'
    and s.plan_id not like '%analyze%'
  where e.experiment_id = 'eid20_premium_1user_35dollars_payday'
"
  
# query BQ
users <- dbGetQuery(con, sql)

There are around 38K users that have been enrolled in this experiment.

Exploratory Analysis

First let’s look at how many users are on each plan. The list below displays users current plans. There are a couple interesting things to note:

• There are a bunch of Pro users in these groups. That’s because we enrolled Pro customers and Pro trialists in the experiment, which means that their subscriptions could have begun before we started the experiment. We’ll end up excluding them from the analysis.

• There’s one customer on the $35/mo Premium plan that got enrolled in the control group. After investigating, I found out that this user is Char. We’ll remove her user from the dataset.

• There are users on Free and legacy Awesome plans that were enrolled because they had started Pro trials.

Effect on Total Revenue

Now let’s assign MRR amounts to each plan id and estimate total MRR for each group. The code for assigning MRR amounts to plans has been hidden.

This table includes all users enrolled in the experiment, including Pro users. We can see that total MRR and ARPU are slightly higher for users in the enabled group.

Effect on Premium and Business Revenue

Because users could have been on the Pro plan before being enrolled, it might be more useful to only look at users on Premium and Business plans. We’ll do that in the table below. The code has been hidden again.

As you can see, Premium and Business MRR is significantly higher for users in the enabled group. This validates the hypothesis that offering a lower priced Premium plan would result in more customers subscribing to Premium.

It’s worth noting that ARPU is lower for users in the enabled group, however that is more than offset by the presence of more customers that subscribed to Premium and Business plans.

Effect on Premium

Let’s narrow our focus and look only at the Premium plans. We can see that more people in the control group subscribed to the $65 Premium plans, however this difference is compensated for by the number of customers that subscribed to the $35 Premium plan.

Total Premium MRR is higher in the enabled group ($2804 compared to $2483), even though ARPU is lower ($43.13 compared to $65.34).

Subscription Started and Upgraded Events

Let’s gather these events from Mixpanel. These are all of the subscription events in Mixpanel that were triggered by users assigned to an experiment group.

library(RMixpanel)

# set up project
account = mixpanelCreateAccount("Buffer",
                                token = Sys.getenv("MIXPANEL_TOKEN"),
                                secret = Sys.getenv("MIXPANEL_SECRET"), 
                                key = Sys.getenv("MIXPANEL_KEY"))

# get enrollment events
enrollment_events <- mixpanelGetEvents(account,
                            where = 'properties["experimentId"] == "eid20_premium_1user_35dollars_payday"',
                            from = "2019-10-17",
                            df = TRUE,
                            verbose = FALSE)

# get subscription events
subscription_events <- mixpanelGetEvents(account,
                            event = list("Subscription Started",
                                         "Subscription Upgraded",
                                         "Subscription Downgraded"),
                            from = "2019-10-18",
                            to = "2019-11-01",
                            df = TRUE,
                            verbose = FALSE)

# join events
events <- subscription_events %>% 
  inner_join(select(enrollment_events, distinct_id, experimentGroup), by = "distinct_id") %>% 
  select(event, email, distinct_id, event, experimentGroup, planId, previousPlanId, product) %>% 
  filter(product != "analyze")

Let’s look at the types of events that were returned.

# count event types 
events %>% count(event, sort = TRUE)

## # A tibble: 3 x 2
##   event                       n
##   <chr>                   <int>
## 1 Subscription Started       83
## 2 Subscription Upgraded      31
## 3 Subscription Downgraded    15

Wow, it looks like there were 15 downgrade events to the new premium plan. Let’s look at the plans that users downgraded from.

# look at downgrade events
events %>% 
  filter(event == "Subscription Downgraded") %>% 
  count(previousPlanId, sort = TRUE)

## # A tibble: 4 x 2
##   previousPlanId                  n
##   <chr>                       <int>
## 1 pro_v1_yearly                  10
## 2 premium_business_v1_monthly     2
## 3 premium_business_v1_yearly      2
## 4 publish_premium_m_35_201910     1

Two users downgraded from the yearly Premium 35 plan, and ten downgraded from the Pro Yearly plan. Those should really be upgrades eh. Let’s change the type of event for those events.

# switch downgraded events to upgraded
events <- events %>% 
  mutate(event = ifelse(event == "Subscription Downgraded" & 
                          (planId == "publish_premium_m_35_201910" | 
                             planId == "publish_premium_y_336_201910") &
                          previousPlanId == "pro_v1_yearly", "Subscription Upgraded", event))

Now, let’s look at all the subscription started and upgraded events for users in each group.

## # A tibble: 9 x 3
##   planId                       control enabled
##   <chr>                          <int>   <int>
## 1 pro_v1_monthly                    32      31
## 2 pro_v1_yearly                      8      11
## 3 publish_premium_m_35_201910       NA       5
## 4 premium_business_v1_monthly        4       3
## 5 business_v2_small_monthly          2       2
## 6 premium_business_v1_yearly         4       2
## 7 publish_premium_y_336_201910      NA       2
## 8 pro15_v1_monthly                  NA       1
## 9 business_v2_small_yearly           1      NA

Looks like we may be missing a couple Mixpanel events here, but still useful to see. :)