Setup for basic A/B tests for the front page

[webmaven]

Currently, the front page has around an 80% bounce rate, possibly because it is targeting receivers. I propose testing our current Hypothesis for 'Customer A' by running an A/B test on the front page content to see if we can move that needle significantly.

To set up a basic A/B test We'll need the following steps completed as prerequisites

• @whit537 needs to sign up for Optimizely, their cheapest plan should be fine (30 day free trial, then $19 per month after trial period)
• The Optimizely JS code needs to be included into the base template, or specifically into the front-page template: https://help.optimizely.com/hc/en-us/articles/200040095-Implementing-the-Optimizely-Snippet
• Turn on Optimizely integration for Segment: https://segment.com/docs/integrations/optimizely/
• Turn on Optimizely integration with Google Analytics
• Agree upon, and state a hypothesis we will try to validate
• Give access to Optimizely to the person who will set up the experiments
• BONUS Add a 'thank you for signing up' page for people signing in for the first time Didn't make it.
• Set up one or more variations to run on the front page, including tracking the events we are trying to promote (signing in using any of the buttons)
• turn on the experiment, wait for statistical significance and a bit more (shouldn't take more than a few days, perhaps as little as one day)
• turn off the experiment, and evaluate the results

Note: I am not certain that the variations will end up in GA (Google Analytics) so that we can limit the analysis to first time visitors and segment by variation, but even if not, we should see a statistically significant change as first time visitors currently bounce at a ~75% rate., and we can also see Optimizely's own reports.

[techtonik]

Are there bounce rate stats on Gittip name and previous designs of the front page? For me the page is definitely not "scannable" - it is hard to say what it is about without reading thoroughly.

[webmaven]

Proposed text changes for a first experiment (aimed at targeting our Customer A):

"Weekly Payments" -> "Weekly Donations"

"Motivated by Gratitude" -> "Motivated by Gratitude and Love"

"Sustainably crowdfund your business, personal projects, or charity, with no extra fees."  -> "Donate anonymously to people you are grateful to, admire, or love, with no extra fees."

"to Accept Weekly Payments" -> "To Give Weekly Donations"

The faster we want results of an experiment, the fewer variations we should test. I propose we test one variation with all these (or similar) changes against the current default. We could definitely try a dramatic design change instead (or in addition) but the key is that we test one variation on the page (not just one change), as we otherwise won't have enough first-time traffic to get results in a reasonable amount of time.

BONUS: We'll also see any effect on returning users but that isn't what we're trying to test for here.

[webmaven]

@techtonik Current bounce rates are available in GA, yes, so if you have dates for design changes, I can tell you if they seem to have made a significant difference, but without split testing, confidence will be limited.

[rohitpaulk]

👍

[chadwhitacre]

Funnel:

1. be interested in what we have to offer
2. hear about us
3. visit
4. read about
5. explore examples (find someone to give to?)
6. sign up
7. enter credit card info
8. find someone to give to (if haven't yet)
9. pick an amount

[chadwhitacre]

IRC

[chadwhitacre]

@webmaven May I give you access to Optimizely?

[chadwhitacre]

IRC re: Segment + Optimizely

[chadwhitacre]

Looks like we're going to use Optimizely directly instead of through Segment, because of price.

[chadwhitacre]

New visitor bounce rate:

[webmaven]

@whit537 Yes, please give me access.

[chadwhitacre]

@webmaven I've attempted to do so via LastPass. Please let me know if you don't get an invite or are otherwise stymied.

[webmaven]

Current analytics on new user front page drop off for the last week:

[webmaven]

We're looking for dramatic changes in behavior if we can find them, but small ones add up too.

Hypothesis: Changing the content (see #101 (comment)) of the front page to appeal to givers rather than receivers will not measurably change the new visitor bounce rate.

This is the smallest (and simplest) possible experiment I can think of that might have the desired effect. We can, instead, make a more dramatic change to the page.

Depending on whether the behavior change is dramatic, it may take a while for statistical significance to accumulate (the smaller the effect on user behavior, the more samples will be needed to be certain the effect is there).

We may see other behavior changes in the data that will be worth following up such as a change in which action is taken next.

Note: Unless the results show a rise in bounce rate or other clearly undesireable effect, I suggest the text change be made anyway (first, avoid harm) as it better aligns with current goals.

Possible followup experiments on the front page: Changing the layout, Text sizes, arrangement of elements, color-scheme, Call to Action, etc.

Other possible pages to experiment on can be the 'about' page, adding a 'thank you for signing up' page, changing the un-logged-in view of the receiver profile page to appeal more directly to givers, etc.

[rohitpaulk]

I'm excited to see what happens with this!

[chrisdev]

👍

[chadwhitacre]

IRC

[chadwhitacre]

"Weekly Payments" -> "Weekly Payments"

"Motivated by Gratitude" -> "To People You Love and Believe In"

"Sustainably crowdfund your business, personal projects, or charity, with no extra fees."  -> "Donate anonymously to people you love and admire, with no extra fees."

"to Accept Weekly Payments" -> "to Give Weekly Donations"

[chadwhitacre]

Our first A/B test is live! 💃 Went with:

"Weekly Payments" -> "Donate Weekly"

"Motivated by Gratitude" -> "to someone you love and admire."

"Sustainably crowdfund your business, personal projects, or charity, with no extra fees."  -> "Anonymous. No extra fees."

"to Accept Weekly Payments" -> "to Give Weekly Donations"

[chadwhitacre]

I'm seeing the variant:

[chadwhitacre]

Using 25% as current CTR, here are some sample sizes per Optimizely's calculator:

• 1,000 for 20% effect
• 3,750 for 10% effect
• 15,000 for 5% effect

We're getting about 1,500 new visitors per week.

[chadwhitacre]

sigh

Sorry for the "close" noise. :-/

[chadwhitacre]

Flub:

[webmaven]

What's the flub?

[chadwhitacre]

"Welcome back to Give Weekly Donations!"?

[webmaven]

What did it say before?

[rohitpaulk]

@webmaven - It used to show "Your balance is XXX"

[rohitpaulk]

Looks like the selector used is .action p. What options does Optimizely provide to target elements?

[chadwhitacre]

@rohitpaulk @webmaven How about if we only use Optimizely with anonymous visitors: gratipay/gratipay.com#2893?

[webmaven]

I don't like that option. Can we instead change the CSS class on the p elements (p.greeting vs p.subhead, perhaps)?

If that change to the templates is deployed, I can target the Optimizely rule more narrowly.

Alternately, just change the class of the greeting from div.action to div.greeting to accomplish the same thing, and I may not even have to change the rule.

[chadwhitacre]

I don't like that option.

Why? It's ready to go and solves the problem for the time being. The purpose here is to run one single A/B test to get a feel for how this all works. Let's not overthink this.

[webmaven]

Because I want to see the effect of the new language on logged in users as well.

[webmaven]

I'm on my way back home at the moment. In about an hour I should be able to give you a glimpse at what the analytics look like inside of Google Analytics for this test.

[webmaven]

Optimizely is not set up to distinguish between logged-in and logged out users. It isn't even distinguishing between new and returning users.

But in Google Analytics, we will be able to slice and dice the information much more finely, as long as we don't throw it away.

About 1/4 of the user sessions recorded yesterday (GA has a day's delay) were returning users, and presumably some of those are already logged in. If we stop showing the experiment to logged in users, we'll have invalidated a huge chunk of our data so far, and be very limited in the kinds of questions we can expect to get out of the data after the experiment is concluded. I would really rather not cut ourselves off from potential insights.

[webmaven]

Alternate fix for the flub (untested): https://github.com/gratipay/gratipay.com/pull/2894/files

[chadwhitacre]

Flub fixed in #2894. IRC

[chadwhitacre]

First of three goals is done!

Starting to discuss results and next steps in IRC.

[chadwhitacre]

Or not done!

http://optimize.ly/~tbMtZt?token=b25424135e86a9baef08#view=2

[chadwhitacre]

Per above we need 15,000 users to confidently measure a 5% increase. We get 1,500 new users a week so it would take us another nine weeks.

[webmaven]

I think it is going slower than it could, because we're only able to test users that aren't blocking 3rd-party JS (most ad blocker software, for example, will prevent Optimizely from working). We're pretty sure at this point that the new version isn't worse at least.

I suggest we let this run about another week at most. Meanwhile, no need to let this stop us from furthering the A/B infrastructure and doing other tests in parallel.

[chadwhitacre]

@webmaven Are we tracking only new visitors on Optimizely?

[webmaven]

No. That was why the GA integration was important, so we could differentiate between new and returning visitors there.

[chadwhitacre]

Ah. So in that case the percentages we're getting from Optimizely are meaningless, right? We are interested in the effect of this change on the bounce rate for new visitors, correct?

[webmaven]

So in that case the percentages we're getting from Optimizely are meaningless, right?

Not completely meaningless, it was important to note whether there was an effect in general.

We are interested in the effect of this change on the bounce rate for new visitors, correct?

That is our primary interest, yes.

[webmaven]

Some data so far:

Among new users who land on the front page (1,242 users), GA says the new version has a 53.8% drop off compared to the original of 55.5%.

Among new users that land on another page but then proceed to the homepage (228 users), the new version has 51.3% compared to 51.2% for the original.

If the homepage is the 3rd page visited (129 users), we see the new version with 48% drop off, and the original with 45%.

[chadwhitacre]

"Win The Internet With A/B Testing"
https://hackdesign.org/lessons/44

[chadwhitacre]

One down! :-)