OPTIMIZATION
How to Measure the Impact of an A/B Testing Program
Mat Vermilyer
Director, Data & Analytics
OPTIMIZATION
How to Measure the Impact of an A/B Testing Program
Mat Vermilyer
Director, Data & Analytics
While A/B testing remains one of the best tactics to gain customer insights, improve user experience, increase conversions, and lower acquisition costs, being able to quantify the value of your overall A/B testing program can often be challenging.
8 Steps to Measure the Impact of an A/B Testing Program
Having conducted tens of thousands of A/B testing programs for clients, PeakActiivty developed the following 8-step process to calculate the overall impact or value associated with your A/B testing program.
To begin, you must have already pulled basic data from your test that will serve as the basis for the formulas moving forward. For the purposes of this tutorial, we’ve provided you with sample data that will be used throughout the examples below.
Control Treatment:
Number of sessions = 62,000
Revenue generated = $2,081,976
Test Treatment:
Number of sessions = 62,754
Revenue generated = $2,181,976
Step 2: Calculate Incremental Lift in Revenue Per Session
This step will tell you the incremental revenue gained per session by running the winning experiment.
Formula:
Subtract revenue per session of the control from the test treatment. Then, divide that number by the revenue per session of test treatment and multiply the answer by 100.
Example:
$34.77 – $33.58 = $1.19 incremental revenue per session ($)
$1.19 ÷ $34.77 x 100 = 3.54% incremental revenue per session (%)
Step 3: Calculate the Cost of Running the Test
If the sessions that viewed the control treatment had received the test treatment instead, we would expect that those sessions would also generate 3.54% in incremental revenue. However, since we ran a split 50/50 test, 50% of sessions did not garner incremental revenue. Therefore, there was a cost to run the test.
Formula:
Multiply revenue generated for control treatment by the incremental revenue per session (%).
Example:
Control Treatment: $2,081,976 x 3.54% = $73,783 cost of test
Step 4: Identify a Multiplier for Test Distribution
A multiplier is needed to estimate the value of the winning experiment once rolled out to 100% of traffic. Use the following table to identify your multiplier based on how you’ve split your traffic in your test.
Since our A/B ran at a 50/50 split, we will use a 2.00 multiplier in the remaining calculation.
Step 5: Calculate the Value Gained If Rolled Out 100%
Formula:
Multiply the cost of running the winning test by multiplier.
Example:
Test Treatment: $73,783 ✕ 2.00 = $147,566 value of test at 100%
Step 6: Calculate Value Gained From Testing Period
We now know how much it cost to run this test and the cost to be gained if it were to be rolled out at 100% of traffic. We can, therefore, now calculate the value gained by the winning test only during the testing period.
Formula:
Subtract the cost of running the test from the value gained if rolled out by 100%.
Example:
$147,566 – $73,783 = $73,783 value of testing during testing period
Once a treatment wins and is rolled out at 100%, you can assume that treatment will stay in place and continue to gain incremental revenue for a set period of time. Companies may vary what period of time they use to forecast revenue, but we typically use 120 days as a general benchmark, as incrementality tends to decline after that for a variety of reasons, including seasonality, customer behavior, other A/B tests, and promotional periods.
Formula:
Divide the value of the test at 100% by the number of days that the test ran. Then, multiply the daily gain by the forecasted number of days.
Example:
$147,566 ÷ 35 days = $4,216 daily gain
$4,216 X 120 days = $505,940 forecast incremental revenue gain
Step 8: Calculate Overall Program Value
Your program will likely run multiple tests, and your hypothesis will likely win in some experiments and lose in others. That’s ok. However, to calculate the total value of your A/B testing program, you need to take it all into account.
To do so, you’ll need to complete the above steps for every step you run, winners and losers. You’ll then take that total and subtract the operational cost of running said program. We typically estimate approximately $10K to run a program. Although it will vary depending on factors like staffing, overhead, time period, it can be used as a general benchmark if you don’t already know your own internal operational cost.
Formula:
Add all winning and losing tests together. Then, subtract your operational cost (or $10K benchmark) from the total cost. The new number is the overall value of your A/B testing program.
Investing in Testing
As you begin to run more tests you’ll want to make sure that your testing program is profitable. Leveraging the above formulas will ensure that you are running impactful tests with positive value gains. While not all tests will have revenue per session (RPS) as a primary metric, a similar attribution model can still be used. Remember, no matter what your actual metric for success is, it has to be measured in order to be evaluated.
Want even more educational content?
Subscribe below to get timely content delivered to your inbox,
or fill out the form below to speak to a Peakster about your next project.
DIGITAL MARKETING
Top 10 Photos of Developers Staring At Screens
DIGITAL STRATEGY
5 Ways Retailers Can Make Influencer Marketing More Influential
ECOMMERCE
