Assessing the Effect of the Business Trialist Blank State

On July 12 we made a change to improve the experience for Business trialists that did not have any profiles connected. This was called the “blank state”.

In this analysis we’ll check if the fix had any impact on the proportion of business trialists that were able to connect a profile and schedule a post.

So far, there isn’t much evidence to suggest that the change has made a large impact on the number of trialists connecting profiles. Interestingly though, there is some evidence that the change has had a positive effect on the proportion of trialists that create posts.

Data Collection

Let’s look at Business trialists that started trials on the same day they signed up for Publish. A couple of important choices:

This only includes trialists that started a trial within 12 hours of signing up.
This only includes profiles that were connected within 6 hours of starting the trial.
This only includes posts created within 12 hours of starting the trial.

# connect to BigQuery
con <- dbConnect(
  bigrquery::bigquery(),
  project = "buffer-data",
  dataset = "dbt_buffer"
)

# define sql query
sql <- "
  select
    t.id as trial_id
    , u.id as user_id
    , date(trial_start_at) as trial_date
    , t.customer_id
    , t.converted
    , count(distinct p.id) as profiles
    , count(distinct up.id) as updates
  from stripe_trials t
  inner join publish_users u
    on t.customer_id = u.stripe_customer_id
    and timestamp_diff(t.trial_start_at, u.created_at, hour) <= 12
  left join publish_profiles p
    on p.user_id = u.id
    and timestamp_diff(p.created_at, t.trial_start_at, hour) <= 6
  left join publish_updates up
    on p.id = up.profile_id
    and up.created_at > t.trial_start_at
    and timestamp_diff(up.created_at, t.trial_start_at, hour) <= 12
  where date(t.trial_start_at) >= '2019-04-01'
  and t.plan_id in ('business_v2_small_monthly',
                    'business_v2_small_yearly',
                    'business_v2_medium_monthly',
                    'business_v2_medium_yearly',
                    'premium_v1_monthly',
                    'premium_v1_yearly',
                    'business_v2_large_monthly',
                    'business_v2_large_yearly')
  group by 1, 2, 3, 4, 5
"

# collect data
trials <- dbGetQuery(con, sql)

There are around 16 thousand trials in this dataset.

Effect on Profile Connections

Let’s plot the trials over time.

We can see that there are somewhere around 150 business trial starts per day. Let’s view the number of trialists that were able to connect a profile.

It’s hard to tell what is happening here, so let’s plot the proportion of trialists that connected a profile.

It looks like around 60% of Business trialists connect profiles within 6 hours of starting a trial. There doesn’t appear to have been a large effect on the proportion of trialists that are connecting profiles just yet.

Effect on Posts Sent

Let’s plot the proportion of trialists that create posts in Publish.

It does look like the proportion of trialists that create posts within 12 hours of starting the trial has increased since the change was made, although it is still below 20%.