How to measure the impact of an A/B Testing program

A/B Testing Program Overview

Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.

Problem Statements: 

  • What is the dollar value of an A/B Testing program?
  • How long should we attribute incrementality for “go-forward” winning experience running at 100%?
  • How can I make sure I am running meaningful tests?
  • We are running all these tests but I am not seeing it hit my bottom line. 

WHAT:

This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue.

HOW:

EXAMPLE MATH:

This is a winning A/B test that ran at 50/50 for 35 days. 

STEP 1 – CALCULATE REVENUE PER SESSION

Control RPS = (Control Sales: $2,081,976 ÷ Sessions: 62,000) = $33.58

Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) =  $34.77

STEP 2 – CALCULATE AVG SALES LIFT 

Avg Sales Lift = ( $34.77 ÷ $33.58 ) -1 = +3.54%

NOTE: It’s important that this metric is at statistical significance( >95% confidence).

STEP 3 – CALCULATE THE COST OF RUNNING TEST

Explained: If the control sessions were to receive the test variant we would expect to get the same sales lift that the test variant received. 

Cost of running test = ( Control Sales: $2,081,976 * Avg Sales Lift: 3.54% ) =  $73,783

STEP 4 – CALCULATE MULTIPLIER FOR TRAFFIC SPLIT

Explained: This is more important for multivariate tests. Standard A/B or 50/50 test uses a multiple of 2x. This is needed to forecast what this experience will receive while running at 100%

Total Sessions = 62,000 + 62,754 = 124,754

Control Traffic Distribution = 62,000 ÷ 124,754 = 49.7%

Test Traffic Distribution = 62,754 ÷ 124,754 = 50.3%

Multiplier = 100% ÷ 49.7% = 2.01

STEP 5 – CALCULATE VALUE OF THE CHANGE IF ROLLED OUT 100%

Value = (Cost of running test $73,783 * Multiplier 2.01) =  $148,463

STEP 6 – CALCULATE VALUE GAINED FROM TESTING PERIOD

Value = (Cost of running test $73,783 – $148,463) = $74,680

STEP 7 – FORECAST INCREMENTAL REVENUE FOR EXPERIENCE AT 100%

Test Duration = 35 days

Avg Sales Per Day = ($148,463 ÷ 35) = $4,241

Projection = ($4,241 * 120 days) = $509,018

Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time. 

STEP 8 – CALCULATE OVERVIEW PROGRAM VALUE

Program Value = ( Winning test + Losing tests + Cost )

Winning test: Steps 1-7

Losing tests: Steps 1-3 negative number. 

Cost: We found that it costs ~$8-10K per test after meetings, creative design, test plans, analytics, development, and QA.  For accuracy, we recommend that each program does their own cost calculations.

SUMMARY: 

As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals and more. This can validate that we are running impactful tests with meaningful hypotheses.

Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place. 

Useful excel formulas

  • Statistical Significance

NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )

  • Confidence Interval

 CONFIDENCE.NORM (alpha 0.05, standard dev, size)