The Effect of Business Trials in New Publish

On May 8, we started putting all new-to-Buffer Business trialists in New Publish. In this analysis, we will attempt to detect any significant effect that this change has had on Business MRR. We’ll use the same methods used in a previous analysis of Pro trials in New Publish.

The goal of the Publish team is to have no significant negative impact on Business MRR.

Because there are known issues with trials data that are partially due to our business logic and Stripe implementation, we will focus on upgrades and MRR instead of trial conversions. I believe these are the metrics we care most about, and they will include the trialists.

Data Collection

Let’s start by collecting Business MRR data. We will focus only on Stripe MRR, since New Publish mostly affects Stripe (web) users.

select
  m.date
  , sum(m.mrr_amount) as mrr
from dbt.stripe_mrr as m
left join dbt.plans as p
  on m.plan_id = p.id
where p.simplified_plan_id = 'business'
and m.date >= '2019-01-01'
group by 1

MRR

We can quickly plot how Business Stripe MRR has changed over the past few months.

Wow. It seems pretty clear that there isn’t a negative impact. Let’s plot the daily growth of Business MRR.

We can perform a Causal Impact analysis to see if there is evidence of a negative effect on Business MRR growth. We already know that there isn’t a negative impact, but let’s do it anyway for kicks and giggles.

We run the analysis using the CausalImpact command.

# run analysis
impact <- CausalImpact(mrr_ts, pre.period, post.period, model.args = list(niter = 5000, nseasons = 7))

# plot impact
plot(impact) +
  labs(title = "Impact on Business MRR Growth", 
       subtitle = "There is no evidence of a negative impact on Business MRR Growth") +
  buffer_theme()

As of May 30, there is not sufficient evidence to suggest that there has been a negative impact on Business MRR growth. In fact, there is more evidence towards a positive impact on MRR growth. Let’s view a summary of the model.

# summarize model
summary(impact)

## Posterior inference {CausalImpact}
## 
##                          Average         Cumulative    
## Actual                   530             14308         
## Prediction (s.d.)        46 (163)        1242 (4401)   
## 95% CI                   [-267, 372]     [-7207, 10036]
##                                                        
## Absolute effect (s.d.)   484 (163)       13066 (4401)  
## 95% CI                   [158, 797]      [4272, 21515] 
##                                                        
## Relative effect (s.d.)   1052% (354%)    1052% (354%)  
## 95% CI                   [344%, 1732%]   [344%, 1732%] 
## 
## Posterior tail-area probability p:   0.00203
## Posterior prob. of a causal effect:  99.79654%
## 
## For more details, type: summary(impact, "report")

This summary states that, before Business trials were put into New Publish, Business MRR grew by an average of $38 per day. After we started putting Business trials in New Publish, Business MRR started growing by an average of $530 per day. This represents a 1283% relative increase, and is not likely to have occurred by chance.

We should keep in mind that other changes have occurred around the same time period. We have introduced the Premium plan, launched Instagram first comment, and launched Shop Grid. A combination of all of these factors could be the actual cause of this recent positive trend.