Summary

• Experiment ID: eid10-point-homepage-ctas-to-pricing-page-october-2018
• Traffic Split: 50%
• Length of Experiment: 4 Weeks
• Start Date: Monday, 2018-12-10
• End Date: Tuesday, 2019-01-08
• Analysis Date Range: Tuesday, Dec 11 (SOD) - Monday, Jan 7 (EOD)
• Dashboard: Looker
• Please note, this dashboard will only be visible for Buffer employees

Hypothesis

IF we drive all the homepage signup CTAs to the /pricing page THEN we’ll see a decrease in total signups, an increase in total activated users, and an increase in trial-starts BECAUSE this signup flow will help visitors qualify/self-select into the right plan. We may see less total signups, but out of those signups we’ll see a total higher number of activated users.

The belief behind this: Seeing the pricing page will provide visitors with more context about what plans are available, which will allow them to better select the right plan for their situation. Many people who would have previously just signed up for a free account (because that’s the CTA they clicked from the homepage and we didn’t present them with all our available plans) will choose to start a Pro or Business trial instead.

Experiment Design

• Split all website traffic 50/50 between the control and enabled groups using handlebars
• Run for 4 complete weeks
• Our success metrics are:
• Increase in activated users
• Increase in total trials started
• No significant drop in activated users from Free plan
• Changes to the enabled group:
• All 3 CTAs on the homepage https://buffer.com will be pointed to /pricing (instead of /signup on a Free plan)

Results

* TL;DR: Our hypothesis was supported by the experiment findings

In order to get the most accurate results possible, we filtered the results in the following ways:

1. only included users and visitors where the associated visitor_id was added to the experiment between the start of the day on December 11th and the end of the day on January 7th.
2. only included users created between the start of the day on December 11th and the end of the day on January 7th.
3. only included users that have at least one presignup visit to / or /pricing between the start of the day on December 11th and the end of the day on January 7th.

Learnings

• Redirecting all Hompage signup CTAs to /pricing resulted in more total accounts starting trials (combining during signup and after signup), more activations for signups w/o trials and signups w/ Pro trials, more accounts upgrading to a paid plan, more total business trials converting to a business plan.
  
• CTA tracking showed that signups that would have occurred on the home page were mostly shifted to the pricing page, and that signup patterns from CTAs not on the home page or pricing page were unaffected.
• We suspect that the drop in % of users that start a Business trial after signup is a result of a shift of a portion of those users that would have started after trial to doing so during signup. This does not appear to be a concern at this time, as total number of Business trials started as a combination of before and after signup increased in the enabled group.
• The one possible concern might be the drop in activation rate for users that signup with a Business trial. However, this may be a result of simply getting more visibility to the option to signup with a business trial, and therefore an increase in users signing up via this option that are “just looking around”.
  
• The decrease in total signups were a result of a decrease in signups w/o trials. After comparing this to the total number of trials started and activations, it is clear that the slight drop in total signups and signup w/o trials were related, and this was the result of filtering out lower level of interest users.

Recommendation

• Wait until mid-march to assess upgrade rates
• Implement the treatment or include this flow in the new marketing site design