## How to measure the impact of an A/B Testing program

**A/B Testing Program Overview**

Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.

**Problem Statements: **

- What is the dollar value of an A/B Testing program?
- How long should we attribute incrementality for “go-forward” winning experience running at 100%?
- How can I make sure I am running meaningful tests?
- We are running all these tests but I am not seeing it hit my bottom line.

**WHAT:**

This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue.

**HOW:**

EXAMPLE MATH:

This is a winning A/B test that ran at 50/50 for 35 days.

**STEP 1 – CALCULATE REVENUE PER SESSION**

Control RPS = (Control Sales: $2,081,976 ÷ Sessions: 62,000) = **$33.58**

Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) = **$34.77**

**STEP 2 – CALCULATE AVG SALES LIFT **

Avg Sales Lift = ( $34.77 ÷ $33.58 ) -1 = **+3.54%**

NOTE: It’s important that this metric is at statistical significance( >95% confidence).

**STEP 3 – CALCULATE THE COST OF RUNNING TEST**

Explained: If the control sessions were to receive the test variant we would expect to get the same sales lift that the test variant received.

Cost of running test = ( Control Sales: $2,081,976 * Avg Sales Lift: 3.54% ) = ** $73,783**

**STEP 4 – CALCULATE MULTIPLIER FOR TRAFFIC SPLIT**

Explained: This is more important for multivariate tests. Standard A/B or 50/50 test uses a multiple of 2x. This is needed to forecast what this experience will receive while running at 100%

Total Sessions = 62,000 + 62,754 = **124,754**

Control Traffic Distribution** = **62,000 ÷ 124,754 = **49.7%**

Test Traffic Distribution **= **62,754 ÷ 124,754 = **50.3%**

Multiplier = 100% ÷ 49.7%** = 2.01**

**STEP 5 – CALCULATE VALUE OF THE CHANGE IF ROLLED OUT 100%**

Value = (Cost of running test $73,783** * **Multiplier 2.01) = ** $148,463**

**STEP 6 – CALCULATE VALUE GAINED FROM TESTING PERIOD**

Value = (Cost of running test $73,783 – $148,463) = **$74,680**

**STEP 7 – FORECAST INCREMENTAL REVENUE FOR EXPERIENCE AT 100%**

Test Duration = 35 days

Avg Sales Per Day = ($148,463 ÷ 35) = **$4,241**

Projection = ($4,241 * 120 days) = **$509,018**

Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time.

**STEP 8 – CALCULATE OVERVIEW PROGRAM VALUE**

Program Value **= **( Winning test + Losing tests + Cost )

Winning test: Steps 1-7

Losing tests: Steps 1-3 negative number.

Cost: We found that it costs ~$8-10K per test after meetings, creative design, test plans, analytics, development, and QA. For accuracy, we recommend that each program does their own cost calculations.

**SUMMARY: **

As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals and more. This can validate that we are running impactful tests with meaningful hypotheses.

Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place.

**Useful excel formulas**

- Statistical Significance

NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )

- Confidence Interval

CONFIDENCE.NORM (alpha 0.05, standard dev, size)