# A/B Testing Program Overview

Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.

## Problem Statements:

• What is the dollar value of an A/B Testing program?
• How long should we attribute incrementality for “go-forward” winning experience running at 100%?
• How can I make sure I am running meaningful tests?
• We are running all these tests but I am not seeing it hit my bottom line.

## WHAT:

This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue.

## HOW:

EXAMPLE MATH:

This is a winning A/B test that ran at 50/50 for 35 days.

### STEP 1 – CALCULATE REVENUE PER SESSION

Control RPS = (Control Sales: \$2,081,976 ÷ Sessions: 62,000) = \$33.58

Test RPS = (Test Sales: \$2,181,976 ÷ Sessions: 62,754) =  \$34.77

### STEP 2 – CALCULATE AVG SALES LIFT

Avg Sales Lift = ( \$34.77 ÷ \$33.58 ) -1 = +3.54%

NOTE: It’s important that this metric is at statistical significance( >95% confidence).

### STEP 3 – CALCULATE THE COST OF RUNNING TEST

Explained: If the control sessions were to receive the test variant we would expect to get the same sales lift that the test variant received.

Cost of running test = ( Control Sales: \$2,081,976 * Avg Sales Lift: 3.54% ) =  \$73,783

### STEP 4 – CALCULATE MULTIPLIER FOR TRAFFIC SPLIT

Explained: This is more important for multivariate tests. Standard A/B or 50/50 test uses a multiple of 2x. This is needed to forecast what this experience will receive while running at 100%

Total Sessions = 62,000 + 62,754 = 124,754

Control Traffic Distribution = 62,000 ÷ 124,754 = 49.7%

Test Traffic Distribution = 62,754 ÷ 124,754 = 50.3%

Multiplier = 100% ÷ 49.7% = 2.01

### STEP 5 – CALCULATE VALUE OF THE CHANGE IF ROLLED OUT 100%

Value = (Cost of running test \$73,783 * Multiplier 2.01) =  \$148,463

### STEP 6 – CALCULATE VALUE GAINED FROM TESTING PERIOD

Value = (Cost of running test \$73,783 – \$148,463) = \$74,680

### STEP 7 – FORECAST INCREMENTAL REVENUE FOR EXPERIENCE AT 100%

Test Duration = 35 days

Avg Sales Per Day = (\$148,463 ÷ 35) = \$4,241

Projection = (\$4,241 * 120 days) = \$509,018

Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time.

### STEP 8 – CALCULATE OVERVIEW PROGRAM VALUE

Program Value = ( Winning test + Losing tests + Cost )

Winning test: Steps 1-7

Losing tests: Steps 1-3 negative number.

Cost: We found that it costs ~\$8-10K per test after meetings, creative design, test plans, analytics, development, and QA.  For accuracy, we recommend that each program does their own cost calculations.

## SUMMARY:

As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals and more. This can validate that we are running impactful tests with meaningful hypotheses.

Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place.

### Useful excel formulas

• Statistical Significance

NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )

• Confidence Interval

CONFIDENCE.NORM (alpha 0.05, standard dev, size)