A/B testing overview
What is A/B testing?
A/B testing is about testing how a new user experience changes behavior. The goal is to ACCURATELY measure how one experience compares to another. For example, your checkout button is usually red. If you change the button color to blue, will you get more conversions? Any part of your user experience is testable, if you set things up correctly.
Our testing background
We have run A/B testing for multiple clients from b2b to large enterprise across a variety of tools. Here are some of our top tips to help you get started on your A/B testing journey. We know this stuff inside and out and can help kick start your journey.
What tool to use?
You need a testing tool to do testing correctly. Mostly this is about controlling which users get which experience. Tough to do this on your own. Optimizely is good. VWO is good. Do some research and find a good one for you. If you need to A/B test native apps that is different than web. I recommend Firebase for native app testing.
How to get started
Here are the steps:
- Install/setup the code necessary to get your A/B testing tool working
- Important that this gets set up correctly. No shortcuts as proper set up helps present the test treatments correctly.
- Brainstorm test ideas; start with the parts of user experience that are closely associated with your KPIs
- Prioritize based on potential impact; its ok if you go with your gut feeling
- For your first tests, it is also helpful to pick tests that have a good amount of traffic. If you want to measure the impact to conversions but you only have 5 conversions a month, it may take a while. Instead, pick a landing page that has more traffic.
- Define your Goal Metric; this is the thing you want to improve through the test treatment
- Implement and run first test
- Once you have significant results, analyze test and use those learnings to start the next one!
Basics of A/B testing
You have a control treatment and a test treatment. The control treatment is your existing experience. The test treatment is what you think will be better.
There are different kinds of tests. A/B is just one. You have A/B/N, MVT, multi-arm bandits, etc. Get complicated later. For now, focus on basic A/B tests. Part of what you need to develop is the discipline of testing and running a test through to its conclusion so it follows your plan.
Don’t worry too much about sample size. Most modern tools will help with the analysis once you get enough of the goal metric. It is the difference between the treatments that is most important.
Ground rules
2 weeks and 2 weekends
Not every site will have enough traffic to run a test in 2 weeks. But if you traffic, resist the temptation to run a test faster. Running a test over 2 weeks and 2 weekends provides a wide enough time range that some unique event you might not be aware of will wash out of the results.
Do not get attached to the test treatment
Many people get attached to the test treatment because it was “their idea”. If your idea isn’t moving the needle for your business then don’t put it out there. Often people get attached to the test treatment and will try to justify why the test was wrong or the test treatment will work better in production.
No it won’t.
If you really think so you can make changes and try again. Stay Zen, no attachment.
The cycle of testing
The winner of one test is the control of the next. Tests should not exist in isolation. What you learn from one test drives the next. And even if a test is a tie you still learn something from that result.
As you progress, you can plan the tests so they work together to create a continuous learning and improvement cycle. Over time, the wins from tests really add up.
Contact us if you would like to explore or learn more about testing.