A/B Testing Program Overview
Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.
- What is the dollar value of an A/B Testing program?
- How long should we attribute incrementality for “go-forward” winning experience running at 100%?
- How can I make sure I am running meaningful tests?
- We are running all these tests but I am not seeing it hit my bottom line.
This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue.
This is a winning A/B test that ran at 50/50 for 35 days.
STEP 1 – CALCULATE REVENUE PER SESSION
Control RPS = (Control Sales: $2,081,976 ÷ Sessions: 62,000) = $33.58
Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) = $34.77
STEP 2 – CALCULATE AVG SALES LIFT
Avg Sales Lift = ( $34.77 ÷ $33.58 ) -1 = +3.54%
NOTE: It’s important that this metric is at statistical significance( >95% confidence).
STEP 3 – CALCULATE THE COST OF RUNNING TEST
Explained: If the control sessions were to receive the test variant we would expect to get the same sales lift that the test variant received.
Cost of running test = ( Control Sales: $2,081,976 * Avg Sales Lift: 3.54% ) = $73,783
STEP 4 – CALCULATE MULTIPLIER FOR TRAFFIC SPLIT
Explained: This is more important for multivariate tests. Standard A/B or 50/50 test uses a multiple of 2x. This is needed to forecast what this experience will receive while running at 100%
Total Sessions = 62,000 + 62,754 = 124,754
Control Traffic Distribution = 62,000 ÷ 124,754 = 49.7%
Test Traffic Distribution = 62,754 ÷ 124,754 = 50.3%
Multiplier = 100% ÷ 49.7% = 2.01
STEP 5 – CALCULATE VALUE OF THE CHANGE IF ROLLED OUT 100%
Value = (Cost of running test $73,783 * Multiplier 2.01) = $148,463
STEP 6 – CALCULATE VALUE GAINED FROM TESTING PERIOD
Value = (Cost of running test $73,783 – $148,463) = $74,680
STEP 7 – FORECAST INCREMENTAL REVENUE FOR EXPERIENCE AT 100%
Test Duration = 35 days
Avg Sales Per Day = ($148,463 ÷ 35) = $4,241
Projection = ($4,241 * 120 days) = $509,018
Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time.
STEP 8 – CALCULATE OVERVIEW PROGRAM VALUE
Program Value = ( Winning test + Losing tests + Cost )
Winning test: Steps 1-7
Losing tests: Steps 1-3 negative number.
Cost: We found that it costs ~$8-10K per test after meetings, creative design, test plans, analytics, development, and QA. For accuracy, we recommend that each program does their own cost calculations.
As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals and more. This can validate that we are running impactful tests with meaningful hypotheses.
Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place.
Useful excel formulas
- Statistical Significance
NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )
- Confidence Interval
CONFIDENCE.NORM (alpha 0.05, standard dev, size)