3 min read

Data Digest February 2019

Welcome to the February 2019 edition of our monthly data digest!

Experiments and Rollout

New Publish

The Publish team continues to make New Publish a more attractive and reliable offering. By the end of February, close to 90% of all active Buffer users were able to access New Publish. Over the next couple of months, the Publish team will continue enticing users to switch from Buffer Classic with new features and other improvements.

As more and more value is added to New Publish, we can track the proportion of active paying customers that are on New Publish with this dashboard.

We can segment those percentages by the user’s Stripe plan ID, their Buffer production plan (e.g. awesome, pro, small, business, agency, etc.), whether or not they are active, and whether they have used the app switcher tool. If you would like to explore this data more, please reach out to Julian or any of the data team members.

Reply Login Flow and Target Customer Survey

The Core team, the Reply team, and Roy have succesfully rolled out a new login flow and the target customer survey to 100% of new Reply signups. This was an impressive undertaking and they all deserve many snaps.

Due to some complications with the experiment’s design and implementation, we couldn’t directly measure the effect that the new login flow, including the survey, had on Reply onboarding and trial starts. However, we could use a technique called causal impact to estimate the effect with the data we had.

The team didn’t allow itself to be blocked and was able to get this out to all signups quickly.

The full causal impact analysis can be found here on the data blog.

The main takeaway is that the new login flow has had no detectable negative impact on Reply onboarding or trial numbers.

Premium Pricing Experiment

Another crossfunctional team has been running an experiment to measure the impact of a new Premium plan offering that closes the gap between the $15 Pro plan and $99 Small Business plan. The planning document for this experiment can be viewed here in Paper, and the data from the experiment can be viewed in this dashboard.

We have stopped enrolling users into this experiment, but we still need to wait for Business trials to finish up. We will update you with the results when the experiment has concluded!

Analysis

We were able to do a couple of focused analyses in February.

The Business Trial Wizard

We ran a very short analysis on the Business trial progress tracker. We found that usage of this feature was highly correlated with the trial conversion rate, suggesting that either the progress tracker helped trialists get their accounts set up and made them more likely to convert, or that successful trialists were more likely to make use of the feature.

The Publish team used this data to make a data-informed decision on whether or not to build this feature for New Publish customers.

When do upgrades happen

We also ran an analysis on the amount of time it took existing Free users to upgrade from New Publish and Buffer Classic. You can read the full analysis here.

Tidbits

When users use the Instagram reminders feature, we have a mechanism to collect the status id from Instagram’s API and associate it with the reminder’s update id. We can use this Instagram status id to infer wehther a user actually posted on Instagram from their reminder. Based on this data point, we estimate that the percentage of reminders that were actually posted to Instagram in the past month is around 73%.

Around 2.4% of respondents to the target customer survey in the Reply login flow have been target customers.

Thanks and see you all next month!