A/B testing is something you need to be doing right now—especially when small improvements of say 1% or 2% can produce impressive revenue improvements down the line.
What Is A/B testing?
Before we get into the specifics of an amazing A/B testing framework, let’s just take a step back and ensure we are all on the same page when talking about A/B testing in general. A/B testing in digital marketing and conversion optimization, also spelled as AB testing, is the process of testing two versions of a webpage or app with controlled differences. The pages are presented to users randomly. As performance data is collected, the pages are analyzed to determine which version performs better.What Are the Benefits of A/B Testing?
Testing variations of a website or app with statistical rigor allows you to measure and optimize your page to ensure you are maximizing conversions. Whether you’re testing a change to the header or just the color of a button, don’t rely on theory to guide these decisions, test it and allow the data to drive decision making. Let’s do some quick math to paint a picture of the impact it can have. Imagine your product sells at $100 and you have 10,000 visits a month to your webpage. 3% of them convert, generating $30,000 in revenue for that one product. You A/B test some changes to the page, and your test shows you that your new page version improves your conversion rate by 3% (so now you’re converting 3.09% of visitors). Less than 0.1% additional conversions doesn’t sound like much, but you’ve just generated an additional $900 a month in revenue. Even if you stop testing at that point, you’ve added $10,800 annual revenue. If you keep testing, and test two more times with the same result of a 3% improvement in conversion rate, your conversion rate goes up by 9.3% (1.03 * 1.03 * 1.03). It’s a lot like compounding interest. Your first month at the new conversion rate will have an incremental conversion value of $2,782. Over a year, this will result in an incremental $33,382. In this particular example, the testing led to more than a month’s worth of incremental revenue. Please note that this is done with the traffic remaining constant. You didn’t invest into higher volume of ads, more content distribution emails, site-wide SEO, or anything else that would grow your traffic. You’re just getting more value out of the traffic volume you’re already getting. You’re optimizing for $-value of each visitor.4 Types of A/B Tests That Deliver Results
There are numerous metrics and even areas of data to explore. Here are 3 ways we can employ split testing to ensure we have the best conversion rates possible.Pricing Experiments
There used to be a marketing theory that prices ending in 7 actually converted better than any other number. Why? Because having a price that finishes with 0 (zero) has been proven to be too general, not specific enough, 9 became commonly used. Then prices ending in 9 became so common that it doesn’t stir up a notable positive response anymore, and doesn’t impact a conversion rate. So why do you think you’re seeing prices ending in 7 now? The 7 was originally proven with an A/B test, where pricing was tested to find the best optimization point. That is what the data suggested, and you can still see it in some SaaS pricing. This may not be the right recommendation for your website and your audience. It’s a good idea to run your own tests, and find the number that best converts for you in particular. Use best practice and industry knowledge to guide what you try, but let your own A/B testing data drive the decision. On a side note, one related use case for AB testing is optimizing for drop-off steps along the check-out funnel. We love using Amplitude Experiment for that.Color Theory
Another example of testing A/B versions of a website or specific page, is the color of the CTA. Yellow was a popular favorite for a long time. Again, thanks to A B testing, data showed that yellow buttons converted better. Since then, the industry has tested colors more. Data from more in-depth testing suggests that it’s more related to the contrast between background and CTA. Not necessarily yellow just how eye catching the CTA is within the context of the page. But, it’s always a good idea to test variations and see what works for your specific CTA in the context of your specific branding.Mouse Tracking
Something as simple as tracking the journey of the user on a webpage can lead to great actionable CRO data. Where do visitors focus their time and attention? Where do they get to on the page when they decide to leave? There are a variety of mouse tracking and heat map software that show you where exactly on the page your user spends most of their time, and where they drop off. By testing the user’s interaction with your website in this way, you can allow the data to determine feature layouts, CTA placements, etc. With this type of knowledge, you are able to make educated decisions to design your tests and measure the results of those changes.Heuristic Analysis
It’s not enough to design a good looking web page or app; the page also has to persuade the audience to complete the desired action. Heuristics are empirical rules of thumb or best practices that have been tested in the past and are implemented to produce UX designs. However, each page, product and service is unique. Rather than just ensuring your page fits with industry best practice, i.e. best practice that comes from somebody else’s testing, it makes sense to test elements such as your call to action placement and conversion rates in real time. Let the design of your website be data driven.The Importance of Using an A/B Testing Framework
You can’t just rush into A/B testing. A haphazard approach will lead to failure. If you’re like most companies, you only have so many resources and so much money, so you have to learn to prioritize — sometimes ruthlessly. If you don’t, your growth will eventually stagnate, or you’ll start shrinking. A/B testing is the fastest, easiest, and most cost-effective way to discover how to drive more traffic, generate more leads, create more sales. If you want to succeed with A/B testing, you need to develop a strategy and priorities ahead of time—which can be tricky to do. Here at McGaw.io, we’ve designed something we call the VICE A/B testing framework. Using this framework, we’re able to avoid a lot of the common A/B testing failures that are caused by lack of strategy and organization. In this post, we’re going to go through the VICE framework and break down exactly how it works. We’ll explore how you can use the VICE approach to maximize your goals (like revenue or user sign ups) without having to spend more money on driving traffic. By the end, you’ll understand how to use this framework in your own business so that you can benefit from all that A/B testing has to offer.The Importance of Creating Hypotheses
If you take a look at the McGaw.io blog, you’ll see that we’ve already covered the basics of how you can get started with A/B testing, like what to test and what tools to use. The focus of this post is to explore how you can conduct A/B tests that are more likely to produce beneficial results—and the key to achieving that is our VICE Framework and Template for A/B Testing. Before you dive into any A/B testing, you first need to form some hypotheses. Hypotheses are essentially what you think will happen as a result of making certain changes. When using the VICE Framework, we leverage hypotheses as our starting point. Generating hypotheses is important because without them, your experiments will lack direction. It’s a good idea to come up with a wide range of hypotheses ahead of time. Having a variety of hypotheses lined up will improve your ability to quickly implement tests in the future, increasing the rate at which you can raise website conversions. You should create at least one hypothesis for each element of a page you want to test. Pro Tip: Refrain from A/B testing more than one element of a page at any given time. Running overlapping A/B tests will create confusing data and poor results. Testing multiple overlapping variations is a “multivariate test,” which is a different sort of test design than a typical A/B split test. Pro Tip: Though it can be helpful to have some background knowledge regarding what works well when running tests, you don’t always have to follow best practice. In some cases, you may find that making radical changes and going against best practice actually leads to better conversions.The VICE A/B Testing Framework
When it comes to A/B testing, it’s important to get some quick wins under your belt. When you take care of low hanging fruit, you’re in a better position to spend time on the more challenging experiments you want to dig into. Plus, the sooner you have some winning A/B tests, the more you’ll gain from the compounding improvements that come with improving conversions. Pro Tip: If you’re doing client work, generating quick wins can be even more important, since it gives you the opportunity to prove that A/B testing is worth focusing on. With that in mind, how can you identify the low hanging fruit and determine the experiments or hypotheses that are most likely to work? Sure, a crystal ball might be helpful here, but odds are you don’t have one. Nevertheless, here at McGaw.io, we think we have the next best thing—the VICE approach. VICE stands for:V – Velocity I – Impact C – Confidence E – Ease
The idea here is that you score your hypotheses from 0–10 (10 being highest/most preferable), in relation to each of the above factors. You then tally up these scores so that you have a “total.” Get the VICE Framework template here! This total helps work out the likelihood of a hypothesis or experiment producing results when compared to all of the other tests you want to run on a site or page. If you’re looking to generate some quick wins, it can be helpful to prioritize the tests that have the highest scores. Here’s an overview of what our framework looks like in action, with the VICE section highlighted: If we zoom in and take a closer look, you can see how we form hypotheses, score each category, and tally up the totals. Each of these categories represent different factors that you need to account for when running an A/B test. Let’s take a closer look at each one so you can give it as accurate a score as possible.Velocity
Velocity is the speed at which a test can be run.
Your velocity score should be based on two main factors: (1) how much traffic a page is getting and (2) how noticable a change will be.How Much Traffic
In general, the more traffic a website is bringing in, the faster a test can be run. More traffic means more data, and more data means it’ll take less time to get a testing sample that’s statistically significant (something we’ll cover later). When we score for Velocity, we integrate with Google Analytics to get the most accurate number possible (want us to integrate your VICE framework into google, contact us.). Velocity is set based upon the amount of traffic that goes through a page. If a page is getting 50,000 people per week, we can hit statistical significance pretty quickly. If a page is only getting 500 hits per week, it will take longer for you to have enough data to be confident in the results.Boldness of Change
Reaching statistical significance isn’t just about how many people come to your website. It’s also about how big or noticable the change you are testing is. The less noticable the change, the less likely it is to impact user behavior, so the longer it will take for your test to reach statistical significance. For example, if you update the navigation text to a bold font, that probably won’t increase the amount of clicks you get dramatically—so it will take longer to reach a level where you can be sure that the change is making a difference. If, however, you make a dramatic change, you can reach statistical significance faster. For example, more users are going to notice a brand new call-to-action button, so you should get enough data to be confident in your results in a shorter amount of time. Think of it like this: if the test is more dramatic or polarizing to the user’s experience, then you should expect the Velocity to be higher because you’ll see a difference faster.Other Testing Factors
It’s also important to look at traffic numbers in relation to how many options there are on a given page. The more options (like links) there are, the longer it will generally take to complete the test. And based on your business or your sales cycle, there are other factors you might need to take into account. For an example, if the sales cycle for one of your products is six weeks long, you won’t be able to complete even a small test in a week. Try to consider your individual business constraints when scoring Velocity. Pro Tip: For the statisticians out there, Velocity can be considered the metric for how long it will take to collect an adequate sample, or how long your test will need to run to have sufficient statistical power.Impact
Impact is how much a change is going to contribute to an improvement in conversions. When you’re considering what Impact score to give a particular hypothesis, it’s important to first think of your goal. If your goal is to increase the use of a free trial, then any changes on the free trial page would get a higher Impact score. If you want to increase email capture to give your sales team more leads, give a higher score to the hypotheses that are likely to impact that area. Once you have your goal in mind, there are three other factors to consider:- The boldness of the change
- The placement of the page in the funnel
- Your analytics data.
Leave a Reply