How to Measure the Impact of an A/B Testing Program

A/B Testing Program Overview

Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.

Problem Statements

What is the dollar value of an A/B Testing program?

How long should we attribute incrementality for the “go-forward” winning experience running at 100%?

How can I make sure I am running meaningful tests?

We are running all these tests but I am not seeing it hit my bottom line.

Benefits of a Positive Workplace Culture

What

This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue. 

“All data has its beauty, but not everyone sees it.”
Damian Mingle

How

This is a winning A/B test that ran at 50/50 for 35 days. 

STEP 1 - Calculate revenue per session

Control RPS = (Control Sales: $2,081,976 ÷ Sessions: 62,000) = $33.58
Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) = $34.77

STEP 2 - Calculate AVG Sales Lift

Avg Sales Lift = ( $34.77 ÷ $33.58 ) -1 = +3.54% NOTE: It’s important that this metric is at statistical significance( >95% confidence).

STEP 3 - Calculate the cost of running test

Explained: If the control sessions were to receive the test variant we would expect to get the same sales lift that the test variant received. Cost of running test = ( Control Sales: $2,081,976 * Avg Sales Lift: 3.54% ) = $73,783

STEP 4 - Calculate multiplier for traffic split

Explained: This is more important for multivariate tests. Standard A/B or 50/50 test uses a multiple of 2x. This is needed to forecast what this experience will receive while running at 100% Total Sessions = 62,000 + 62,754 = 124,754 Control Traffic Distribution = 62,000 ÷ 124,754 = 49.7% Test Traffic Distribution = 62,754 ÷ 124,754 = 50.3%

STEP 1 - Calculate revenue per session

Control RPS = (Control Sales: $2,081,976 ÷ Sessions: 62,000) = $33.58
Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) = $34.77

STEP 5 - Calculate Value of the Change if Rolled Out 100%

Value = (Cost of running test $73,783 * Multiplier 2.01) = $148,463

STEP 6 - Calculate Value Gained from Testing Period

Value = (Cost of running test $73,783 – $148,463) = $74,680

STEP 7 - Forecast Incremental Revenue for Experience at 100%

Test Duration = 35 days Avg Sales Per Day = ($148,463 ÷ 35) = $4,241 Projection = ($4,241 * 120 days) = $509,018 Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time.

STEP 8 - Calculate Overview Program Value

Program Value = ( Winning test + Losing tests + Cost ) Winning test: Steps 1-7 Losing tests: Steps 1-3 negative number. Cost: We found that it costs ~$8-10K per test after meetings, creative design, test plans, analytics, development, and QA. For accuracy, we recommend that each program does their own cost calculations.

Summary

As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals, and more. This can validate that we are running impactful tests with meaningful hypotheses.

Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place.

Useful excel formulas
Statistical Significance
NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )

Confidence Interval
CONFIDENCE.NORM (alpha 0.05, standard dev, size)

Interested In Learning More?

Get your free A/B testing and optimization assessment now!

Sign me up for get a FREE Assestment*