- Experiment ID:
- Traffic Split: 50%
- Length of Experiment: 4 Weeks
- Start Date: Monday, 2018-12-10
- End Date: Tuesday, 2019-01-08
- Analysis Date Range: Tuesday, Dec 11 (SOD) - Monday, Jan 7 (EOD)
- Dashboard: Looker
- Please note, this dashboard will only be visible for Buffer employees
IF we drive all the homepage signup CTAs to the
/pricing page THEN we’ll see a decrease in total signups, an increase in total activated users, and an increase in trial-starts BECAUSE this signup flow will help visitors qualify/self-select into the right plan. We may see less total signups, but out of those signups we’ll see a total higher number of activated users.
The belief behind this: Seeing the pricing page will provide visitors with more context about what plans are available, which will allow them to better select the right plan for their situation. Many people who would have previously just signed up for a free account (because that’s the CTA they clicked from the homepage and we didn’t present them with all our available plans) will choose to start a Pro or Business trial instead.
- Split all website traffic 50/50 between the
enabledgroups using handlebars
- Run for 4 complete weeks
- Our success metrics are:
- Increase in activated users
- Increase in total trials started
- No significant drop in activated users from Free plan
- Changes to the
- All 3 CTAs on the homepage https://buffer.com will be pointed to
/signupon a Free plan)
* TL;DR: Our hypothesis was supported by the experiment findings
In order to get the most accurate results possible, we filtered the results in the following ways:
- only included users and visitors where the associated visitor_id was added to the experiment between the start of the day on December 11th and the end of the day on January 7th.
- only included users created between the start of the day on December 11th and the end of the day on January 7th.
- only included users that have at least one presignup visit to
/pricingbetween the start of the day on December 11th and the end of the day on January 7th.
|CVR % (from Visitors)||3.31%||3.14%||Control is best, statistically significant|
|Total 1+ Profiles||14,217||14,538|
|CVR % (from Signups)||70.15%||72.79%||Enabled is best, statistically significant|
|CVR % (from Signups)||37.53%||39.61%||Enabled is best, statistically significant|
|Signups w/o Trial||19,410||18,558|
|CVR % (from Visitors)||3.17%||2.91%||Control is best, statistically significant|
|Signups w/ Pro Trial||888||1434|
|CVR % (from Visitors)||0.15%||0.23%||Enabled is best, statistically significant|
|Signups w/ Business Trial||509||666|
|CVR % (from Visitors)||0.08%||0.11%||Enabled is best, statistically significant|
|Signups w/o Trial that Activated||7,247||7,340|
|CVR % (from Signups w/o Trial)||37.34%||39.55%||Enabled is best, statistically significant|
|Signups w/ Pro Trial that Activated||190||356|
|CVR % (from Signups w/ Pro Trial)||21.40%||24.83%||Enabled is best, statistically significant|
|Signups w/ Business Trial that Activated||191||227|
|CVR % (from Signups w/ business Trial)||37.53%||34.08%||Draw, no SS, but Control has 89.14% probability of being best|
|Pro Trials After Signup||369||318|
|CVR % (from Signups w/o Trial)||1.90%||1.71%||Draw, no SS, but Control has 91.63% probability of being best|
|Business Trials After Signup||210||158|
|CVR % (from Signups w/o Trial)||1.08%||0.85%||Control is best, statistically significant|
|All Trial Starts||1,476||1,910|
|% of Total Signups||7.28%||9.56%||Enabled is best, statistically significant|
|Upgrades as of Jan 11 (won’t fully capture upgrade impact until mid-March)||423||385|
|% of Total Signups||2.09%||1.93%||Inconclusive, not enough time has passed. We can revisit after 60 days has past from end of experiment.|
Summary of Enabled Treatment
Statistical significant increase in:
- % of signups that connect at least 1 profile
- % of signups that activate
- % of signups w/ Pro Trial
- % of signups w/ Business Trial
- % of signups w/o Trial that activate
- % of signups w/ Pro Trial that activate
- All trial starts (sum of trials at signup and after signup)
Draw (no statistical significance), but chance of drop in:
- % of Pro trial starts after signup
- % of signups w/ Business Trial that activate
Statistical significant drop in:
- % of visitors that signups without a trial
- % of visitors that signup with or without a trial (this is a a result of the drop in signups without a trial)
- % of users that start a Business trials after signup
- Redirecting all Hompage signup CTAs to
/pricingresulted in more total accounts starting trials (combining during signup and after signup), more activations for signups w/o trials and signups w/ Pro trials, more accounts upgrading to a paid plan, more total business trials converting to a business plan.
- CTA tracking showed that signups that would have occurred on the home page were mostly shifted to the pricing page, and that signup patterns from CTAs not on the home page or pricing page were unaffected.
- We suspect that the drop in % of users that start a Business trial after signup is a result of a shift of a portion of those users that would have started after trial to doing so during signup. This does not appear to be a concern at this time, as total number of Business trials started as a combination of before and after signup increased in the enabled group.
- The one possible concern might be the drop in activation rate for users that signup with a Business trial. However, this may be a result of simply getting more visibility to the option to signup with a business trial, and therefore an increase in users signing up via this option that are “just looking around”.
- The decrease in total signups were a result of a decrease in signups w/o trials. After comparing this to the total number of trials started and activations, it is clear that the slight drop in total signups and signup w/o trials were related, and this was the result of filtering out lower level of interest users.
- Wait until mid-march to assess upgrade rates
- Implement the treatment or include this flow in the new marketing site design