How to measure the impact of an A/B Testing program

A/B Testing Program Overview

Below you will find an approach on how to better attribute and measure the impact of an A/B Testing program. If you find yourself asking any of the following questions, throughout your A/B Testing journey then you have come to the right place.

Problem Statements: 

  • What is the dollar value of an A/B Testing program?
  • How long should we attribute incrementality for “go-forward” winning experience running at 100%?
  • How can I make sure I am running meaningful tests?
  • We are running all these tests but I am not seeing it hit my bottom line. 


This document is designed to show the total incremental revenue gained/lost for all tests run. This approach uses revenue per session as the primary metric to determine incremental revenue.



This is a winning A/B test that ran at 50/50 for 35 days. 


Control RPS = (Control Sales: $2,081,976 ÷ Sessions: 62,000) = $33.58

Test RPS = (Test Sales: $2,181,976 ÷ Sessions: 62,754) =  $34.77


Avg Sales Lift = ( $34.77 ÷ $33.58 ) -1 = +3.54%

NOTE: It’s important that this metric is at statistical significance( >95% confidence).


Explained: If the control sessions were to receive the test variant we would expect to get the same sales lift that the test variant received. 

Cost of running test = ( Control Sales: $2,081,976 * Avg Sales Lift: 3.54% ) =  $73,783


Explained: This is more important for multivariate tests. Standard A/B or 50/50 test uses a multiple of 2x. This is needed to forecast what this experience will receive while running at 100%

Total Sessions = 62,000 + 62,754 = 124,754

Control Traffic Distribution = 62,000 ÷ 124,754 = 49.7%

Test Traffic Distribution = 62,754 ÷ 124,754 = 50.3%

Multiplier = 100% ÷ 49.7% = 2.01


Value = (Cost of running test $73,783 * Multiplier 2.01) =  $148,463


Value = (Cost of running test $73,783 – $148,463) = $74,680


Test Duration = 35 days

Avg Sales Per Day = ($148,463 ÷ 35) = $4,241

Projection = ($4,241 * 120 days) = $509,018

Explained: We recommend a 120 day forecast due to the always changing ecosystem. customer behavior, seasonality, promotions, site re-designs, development updates, and further A/B Testing make it difficult to expect consistent incremental lift for more than 120 days. You may also choose to run a winning test at 90/10 or 95/5 so you ensure that the experience is not negatively impacting KPI’s over time. 


Program Value = ( Winning test + Losing tests + Cost )

Winning test: Steps 1-7

Losing tests: Steps 1-3 negative number. 

Cost: We found that it costs ~$8-10K per test after meetings, creative design, test plans, analytics, development, and QA.  For accuracy, we recommend that each program does their own cost calculations.


As we start running more tests we will want to show that the program is profitable. Using this process we can keep track of our win/loss ratio, our incrementality, goals and more. This can validate that we are running impactful tests with meaningful hypotheses.

Not all tests will have RPS as a primary metric however we feel it should still be reported in the same fashion. This attribution model is not meant for all A/B practices however, all A/B practices should have some sort of program overview in place. 

Useful excel formulas

  • Statistical Significance

NORMSDIST(ABS( test avg sales – control avg sales)÷√ ( test std sales^2 ÷ test sessions + control std sales ^2 ÷ control sessions )

  • Confidence Interval

 CONFIDENCE.NORM (alpha 0.05, standard dev, size)

Power Analysis & Sample Size Estimation

Performing power analysis and sample size estimation is an important aspect of experimental design. Without these calculations, the sample size may be too high or too low. If the sample size is too low, the experiment will lack the precision to provide reliable answers to the questions it is investigating. In this case, it would be wise to alter or abandon the experiment. If the sample size is too large, time and resources will be wasted, often for minimal gain.


For each test, we will gather the following four data points before any code or resources are used. 

  1. What is the primary performance indicator (KPI)?
    • By how much do we believe our hypothesis will effect that KPI (effect size)?
      • Our standard can be 1%, 2% ,3%
  2. What is the acceptable minimum confidence level – 90%, 95%, 99% (significance level)?
  3. What is the acceptable minimum power level- 70%, 80%, 90%?
    • Often considered to be between .80 and. 90. 
    • Think of “Power” as the strength of the experiment. Statistical power is the probability that the test will detect an effect that actually exists.
  4. What is the current traffic size on the page being tested? 


With these data points, (effect size, sample size, significance level, power) we can enter three of the four quantities and the fourth is calculated. The basic idea of calculating power or sample size is to leave out the argument that you want to calculate. If you want to calculate power, then leave the power argument out of the equation. If you want to calculate sample size, leave ‘n’ out of the equation. Whatever parameter you want to calculate is determined from the others.


EXAMPLE Power Analysis for – Checkout – Guest Checkout

Hypothesis/Success Criteria: If we clearly call out the guest checkout option then we will increase conversion by at least 2%.

What is the optimal sample size for the given hypothesis? 

  • Sample Size (n)  = Unknown?
  • Effect Size (d) = 2%
  • Power = 80%
  • Sig Level (alpha/confidence level)  = 0.05 or 95%

Sample Size = 19,625

This tells us that we need ~20k sessions to reach 95% confidence to see a 2% increase in conversion at an 80% probability that the detected lift actually exists. If we do not reach a 2% increase in conversion at 95% conf. in the optimal sample size then we failed to reject the null hypothesis.

If we met the 2% increase in conversion rate at 95% confidence in ~20k then we would have rejected the null hypothesis. 

Power Analysis for – Checkout – Guest Checkout – Current Results

What is the Power of our current test results?

  • Sample Size (n)  = 30,946
  • Effect Size (d) = 1.8%
  • Power = Unknown?
  • Sig Level (alpha/confidence level)  = 0.12 or 88%

Power = ~90%

This tells us that there is a 90% probability our test we will be able to detect a change.

However, there is only an 88% confidence level in that change. 

What do we do? 

  • We could accept 88% as “good enough”. 
  • We could re-run our power analysis with a smaller effect size. This will increase the sample size needed. Continue running the test.

Be That Voice in Your Customers’ Heads

How to Brace Yourself for the Voice SEO Revolution

Voice AI assistants are rapidly colonizing consumers’ homes. Amazon’s Alexa voice assistant has infiltrated its way into over 60,000 different gadgets, from your smart speakers to your car. Over forty percent of adults say they use voice search every single day, a prime opportunity for you to market your brand.

Marketing to all these people with all these AI voice options might seem a little tricky. The search results on your Alexa sound a little different from what you might find on regular ol’ Google. But, optimizing your marketing plan for voice search can be critical to your SEO (search engine optimization) success.

Be the answer to every question

Voice search smart assistants Siri, Alexa, and Cortana are making us rethink how we should manage our SEO strategy.

Consider a consumer who asks, “Alexa, is rice gluten-free?” Now, imagine you are a company that has recently developed a gluten-free rice product. You can already see the intent to buy, so all you have to do is put yourself on the other end of that question.

The same question in a text search would probably come up with a list of gluten-free rice brands and brand comparisons. You already know how to set your SEO for text. But the customer asking that question using voice search probably expects a response merely telling them yes or no and suggesting one brand and one price.

The SEO strategy to become that one company listed comes down to how well your content answers direct questions. Voice SEO categorizes by most popular, through continuous testing of conversion rates. Think of it as a constant race to win that one buyer.

Another aspect of voice SEO you may not have taken into account is tone. The tone of voice search is more natural and conversational. This means that your business, your website, and your content should also sound more like that.

It’s like shopping, but without the shopping

Beyond simply marketing your brand, consumers are starting to use voice search to actually purchase your products.

Right now, the options a customer has on a smart speaker consist mostly of hearing what the cart contains and adding and removing products. Walmart recently partnered with Google Assistant to introduce Walmart Voice Order.

So a consumer could order copious amounts of gluten-free rice without ever knowing about your brand. When they order the rice, they probably won’t hear options for a brand or anything else. The assistant simply provides you a “best match” product and brand based on previous purchase habits or recommended items.

So how can you become that “best match” brand? The answer is again voice SEO. Your SEO should adapt to create content that answers the direct search. It’s a winner-take-all game. Once a user has purchased your product, it will begin to populate their recent searches and product recommendations so the likelihood of repeat buys is increased.


Voice SEO should definitely be up your sleeve as part of your marketing bag of tricks. Voice AI is not going away anytime soon as more and more people embrace this technology. Voice search acts as a medium of instant answers and instant eCommerce, so if you’re going to optimize for voice, make sure exact matching is one of your SEO priorities. Take into account your speed and your tone. Users want direct, specific answers and they want them fast.

PeakActivity works hard to create future-proof eCommerce solutions. Voice SEO is the future of commerce and we’re on it. If your customers could benefit from finding you by voice search, we can help you out with that. eCommerce is moving to voice and you need to move with it. Check out our website for more information.

“The Kardashian Effect”

It’s easy to make fun of the often absurd world of social media influencers. While delivering 11 times higher return on investment (ROI) than traditional marketing, influencers have taken over the world, and the companies who are hesitant to jump on the influencer marketing bandwagon are missing out.

The concept of an “influencer” may seem abstract to some marketers, but influencer marketing has become increasingly complex over the last few years. Companies need to know how to simultaneously leverage the right levels of influencers to maximize their brand awareness and drive the greatest results. Here’s how it works.

It’s like getting a recommendation from a close friend who cares.

While some companies have dismissed influencers as superficial or not important, your potential customers have real and authentic relationships with them through social media. Product tutorials and endorsements from trusted sources are influencing people to buy things on social media and influencers carry more weight with customers than anyone could have previously imagined.

Influencers are EXPERTS. They offer deep expertise in a specific area. When a beauty influencer features a makeup brand that she uses every day, her followers are going to trust that she uses “the best products” because they trust her recommendations are authentic. Influencers gain the trust of their followers by bringing them into their inner circle and sharing their lives while using the products.

Influencers are CONNECTED. It’s not always about the number of followers on a platform. Social media influencers, including nano- and micro-influencers, have higher engagement rates of quality followers. This deep level of interactivity builds relationships and is critical to why social influencers are so valuable to any marketing strategy: it’s a platform of trusted fans that facilitates a receptiveness to a brand.

Influencers are AUTHENTIC. The best social influencers are authentic; they have a passion for a brand or product, which shows in what they endorse or recommend to their followers. To their target audience, they are the authority in whatever niche they’re occupying. They’ll tell it like it is with no filter.

Seeing is believing…

Ok, you believe you need influencers, but where do you start? It’s time to find fans of your product or brand. The biggest challenge for most marketers today is finding relevant influencers, people who believe in their brand.

To be successful in influencer marketing, you can’t work with just anyone on social media. You need to find the influencers who would actually believe in your product or service. If they love your product, they will talk about it to their audience in a way that is authentic. If you interact, engage, and help promote them, they will be brand promoters for life.

When looking for influencers, you also want to consider the following: 

  • How do they interact with their fans when not promoting content. What types of comments do they write back to their fans?
  • What is their content style? How do they create content and does it align with where your brand is going?
  • What have you done to engage with them? Influencer marketing is a two-way street, so you can’t expect to get what you aren’t willing to give.

PeakActivity has an incredible social media marketing team that can get in touch with the right influencers for your business. They’ve done it for a bunch of companies with a crazy amount of success. No, for real, they know what they’re doing. Visit our website to learn more.

Alexa, what do you see?

Computer Vision: The next step in artificial intelligence taking over your life

Computers have gotten a little more human-like than anyone is really comfortable with. Robots can open doors, perform surgery, and create literally invincible tic-tac-toe strategies. It’s a crazy world. But one thing computers can’t do? They can’t see you…yet. They actually can’t see anything or understand the context and content of visual things. Artificial intelligence is trying to change that, and it could help to bring in the future of eCommerce.

Turns out, seeing is actually really hard

Can computers see? Most have cameras, but it turns out the answer is a little more complicated than that. Computer vision is the process of giving computers AI algorithms to detect and classify objects more accurately and efficiently. One example of this already in practice is facial recognition. Don’t panic, but computers are probably already identifying your face all the time.

More practical applications of computer vision are the next step for artificial intelligence, and this will allow computers to see AND understand images and videos.  Computer vision has the potential to streamline processes within every department, from manually entering product codes or barcodes in a warehouse to reviewing eCommerce photos.

Computers with vision are probably smarter and faster than you

Computer vision is starting to play a big part in eCommerce. Think of a world where your customer could search for a product with an image of a similar product on your site. Your computer could scour the entire internet for your products and bring back pictures to use on your site.

Even better, computer vision has the potential to take over retail with facial recognition, cashierless stores, inventory visibility, and visual search. This tech has recently made a noticeable impact in these four vital areas. Walmart has already been using computer vision technology in more than 1,000 stores to monitor checkouts and deter potential theft. So maybe don’t steal from Walmart.

Basically, replace all your employees with computers

Computers with vision have the ability to solve tasks that can be time-consuming, from manually entering UPCs or barcodes in a warehouse to reviewing eCommerce environmental photos for specific products. Its expansive use of big data collection allows the software to grow smarter and faster in decision making.

Today, only 3% of retailers have computer vision technology in place, however, 40% plan to start or finish implementing the tech within the next two years. Computer vision is a possibility for any company looking to streamline processes that take a manual user time to visually review and conduct an outcome.

PeakActivity is a leader in creating future-proof eCommerce. If you think your computers need a little bit of vision to drive your company forward, we can help you out with that. Visit our website for more information.

Fire Your Models: Leverage the Average Human Faces Behind Your Products 

User-Generated Content: the marketing goldmine that’s going to make your job easier. 

User-generated content combines your eCommerce website with your customer’s social media pages. By integrating your site to pull images from social media, users can get featured on your website, getting their instafamous glory, while you get totally free authentic marketing content. It’s really a win-win.

Here are three ways that you can incorporate user-generated content into your eCommerce site


Homepage Interactions 

Adding your users to your homepage lets them tell their stories about how great your products are, putting their ratings and reviews right there for everyone to see. Customers get bragging rights about being featured on your website, and new customers are able to see how your products will interact with their day-to-day lives.

One great example is Because everyone is beautiful in their vacation photos, Airbnb actually takes their customers’ photos and testimonies and places them directly on their homepage for all to see. 

Shoppable Galleries 

Shoppable galleries embrace customers who are already tagging brands in their social media photos by making all of those photos feed into a gallery on your website. When potential customers go to the gallery, they can hover over the images to view and purchase the products.

Shoppable galleries will also allow potential customers to see how the product will act in their lives, and encourage brand recognition on social media.

Old Navy’s homepage invites customers to post about them on Instagram. The user-generated content gallery shows real people enjoying Old Navy clothing in their everyday lives, which can make a brand seem more relevant to shoppers.


Everyone loves a good competition, especially if it brings publicity to your brand. Invite your users to participate in your brand by offering prizes, discounts, or features if they post the best pictures of themselves with your product.

Starbucks did this in 2014 with their #whitecup campaign, calling for users to draw art on their white cups and post them on Instagram. The winner would get their art printed on actual cups in actual stores. The contest blew up and tons of artsy Starbucks cups showed up all over Instagram. Starbucks got awesome content to put on their website, and Starbucks customers bought extra coffee just to draw art on the cups.

Our thoughts…

User-generated content is the next big thing. Let your customers write your story for you. After all, they’re the ones out in the world using your products and are probably taking beautiful social media photos anyway. Embrace the world of social media and leverage it into awesome, totally free marketing content.

PeakActivity is a leading expert on eCommerce sites and social commerce, including user-generated content. Leveraging user-generated content to enhance your eCommerce site while connecting to your end user is easy if you have the right people to help you out. Visit our website to find out more and set up a 15-minute call to talk to our team.

Quality Assurance: How It Works and Why It’s Important

The National Institute of Standards and Technology reports that software bugs cost the US economy an estimated $59.5 billion annually.  No one wants defects in their software, but inevitably they always seem to come up, resulting in customer disappointment, negative reputation impact, and revenue risks. This makes quality assurance one of the most important pieces of any software development process.

Quality assurance (usually referred to as QA) is more than a single step of the process; it takes place at every level of the software development life cycle. Testing and quality control help ensure that the development team delivers a high-quality product. Rigorous and systematic testing identifies problems early on in the development process, isolating project risks and maximizing ROI for you and your company.

What Is Quality?

Any software product is created to help its users improve their life experience, satisfy needs, or solve problems, so quality is the ability of a product to meet those objectives. Stakeholders and product owners see quality in an application when it meets their expectations, is delivered on time and within budget, and is maintainable. A software team defines quality as the degree to which the product meets the business requirements.

What Is the Role of the Quality Assurance Team?

The cost of errors in software grows exponentially as they sneak into later phases of the life cycle. To be the most beneficial and identify problems early on throughout product development, QA activities should be part of the development process from the very beginning, even during the planning and analysis stages. This will help reduce risks of creating incomplete or ambiguous requirements, or acceptance criteria that may be unclear on unable to be tested.

Quality assurance focuses not only on finding bugs, but also on preventing software issues, conducting root cause analysis, and educating the team about quality. QA improves the whole software development process, making it more efficient. By evaluating measures and metrics, testing constantly provides very important information about the state of the product. Critical business decisions are made on the basis of that information.

In Agile, which is one of the most widely used and efficient software development methodologies, testing and quality control go hand in hand with code creation. Constant testing allows software teams to find and fix defects in the same phase of the life cycle in which they were introduced, which reduces the risk of producing new defects and shortens product delivery time.

Why Are There Bugs?

Software is created by humans, and human error is inevitable. Defects happen when code is rushed or too complex, but even without these factors, bugs can still appear. While it’s easy to blame the developers, 20% of bugs are created at the requirements level and another 25% at the design level. No one can reduce risks to zero, but quality assurance eliminates critical defects, improves the system, and certifies that the product meets business and industry standards. Along with exit criteria, budget, and timeline, QA testing results factor greatly in any business decisions about a product’s readiness to go live.

Even when a product is released to the public, the job of the quality assurance team is not over. To stay competitive in the market, software should be a constantly changing product that stays highly adaptive to users’ feedback and quickly evolves to keep up with industry standards. Any adjustments or interference with code could potentially bring new issues, and it is always better if they are found by testers than by users.

Our Thoughts…

Building software is a human project, so there will be mistakes along the way, but the quality assurance team gives any project a safety net. The goal of developers is to make sure the product works. The quality assurance team is there to make sure the product works as well as it possibly can.

A high-quality product takes blood, sweat, and tears from every member of the team. A measured approach and a detailed test strategy will ensure a clear understanding of the final result by the customer. QA helps achieve these keys to any product’s success.

PeakActivity has a huge knowledge base in software building, including quality assurance. The right strategy is key to a successful product, and we can help at every step of the way.

Adobe Experience Manager: 5 Ways Your Company Could Benefit  

Your company’s online presence is arguably one of its most important sources of marketing revenue. Having a great online presence requires great content, and great content requires a great content management system.

Adobe Experience Manager (AEM) is one of the fastest growing content management system options for enterprises, used by major corporations like Motel 6 and Chipotle.

One reason AEM has become so popular is that it simplifies the management and delivery of website content while also reducing the complexity of delivering individual online experiences to individual customers.

What is AEM?

Adobe Experience Manager is a content management system that provides one place to create, manage, and deliver websites, mobile sites, or any other kind of application you might need (from kiosks to eCommerce apps). AEM gives you the functionality to personalize the online experience you want and extend it to the tools you need.

What Are the Benefits of AEM?

1) Multiple Sites Managed in One Place

Having everything in one place is one of the greatest benefits of AEM. In the same platform, you can build and deliver web applications for desktop, phone, and tablet, along with any other screens you can think of. Marketing campaigns can be easily launched onto all platforms at the same time. Even if your site has to be in multiple languages and regions, all can be easily managed from the same place.

2) User-Friendly Content Management

Along with having everything available in one place, most sites can be simply created and maintained through easy-to-use templates. Even with limited AEM experience, you can build your own pages and content from the user-friendly dashboard.

3) Adobe Analytics Easily Integrated

Adobe Experience Manager can easily be integrated with Adobe Analytics, one of the best tools to track your website. Reporting visits and measuring page views isn’t enough anymore. Adobe Analytics can measure customer pathing, traffic sources, content effectiveness, and even video engagement. All of this information gives you real data about who is coming to your site and helps you simplify your marketing goals.

4) Specific User Experiences for Specific Users

Along with the data you’ve gathered from analytics, AEM also provides you with multi-site management capabilities. By using Targeting Mode in AEM, you can create a library of user experiences for each page on each platform, a personalization solution that makes it easy to identify and deliver your best content for each user.

See our A/B Testing Article for more information.

5) Adobe Community

The Adobe support community is one of the best for monitoring your site 24/7 and making sure it’s properly supported. If you have your site available in different countries, the support community can even tell you how it’s doing and why. Depending on the size of your company and your support needs, Adobe can even provide you with a dedicated team to make sure that your site is as well maintained as you’d like.

Our Thoughts

Adobe Experience Manager is the clear choice for web content management for many of today’s top companies. With thousands of content management solutions, Adobe Experience Manager has simplified the process by bringing the most important options together in one place. Unlike other CMS options, it can simplify your life by integrating all the important functions of digital marketing and content management onto one single platform.

WordPress vs. Headless Content Management

WordPress vs. Headless Content Management: Choosing Simplicity or Functionality

Reader Takeaways:

  • A content management system (CMS) is a system that stores and displays your company’s content onto a website, mobile app, or another platform.
  • WordPress is great if you’re looking for a simple marketing website you can build without a development team.
  • Headless CMS might be for you if you’re a larger corporation with a team to build custom applications for multiple platforms.

Managing content correctly can make or break your company’s website functionality. With so many new types of content management systems (CMS), it can be hard to decide. In this article, we’ll break down two popular types of content management systems so you can make the right decision.

What Is a Content Management System?

All of the content you’ve created for your website has to be stored somewhere so you can add, remove, and edit when needed. The system can contain a front end, where the content is displayed on a website (or app, game, kiosk, etc.), and a back end, where the data is stored and edited.


WordPress is the simplest, most popular way to create your own website. WordPress is a content management system that is free to use, encompassing both the front end and the back end of your website. That is to say, you add the words and images, pick a theme, and choose some features, and boom… you have a website.

WordPress sites are great all-in-one solutions you can build yourself with preloaded themes and plugins. So WordPress is nice if you have a small website, but it was meant for blogging, so it works best on a site with only information or marketing content.

WordPress has its downsides when it comes to scalability and performance. As soon as you try to add more functionality, like maybe an eCommerce site or a forum for your customers to chat with each other, you’re going to have to create a lot of custom code that could slow down the site.

WordPress might be for you if:

  • You’re looking for an enterprise-level website you can build without a development team.  
  • You need a simple marketing website.
  • You don’t require a lot of functionality or availability on digital devices and platforms.

Headless CMS

While WordPress is your all-in-one solution, a headless CMS has only one focus: the back end. In short, a headless CMS stores your content and leaves the front end display up to you. While that might sound counterproductive, the headless approach can put your content into any technology you can think of, from iOS apps to kiosks to smartwatches and even inside virtual reality headsets.

Using a headless CMS is a powerful option for companies with mobile and web developers. With the right vision, a headless CMS can use any front end tool they want to present content in meaningful and interactive ways.

While a headless CMS can be free to use, just like WordPress, it requires a development team to plan, create, and maintain. And since you’re not working from a template, the user experience can also sometimes suffer. Many growing companies looking for more functionality are choosing headless CMS.

Headless CMS might be for you if:

  • You’re a larger corporation, with a team to build custom applications.
  • You need to publish content on multiple platforms, all at once.
  • A traditional website isn’t doing it for you, and you need more functionality.

What We Recommend

WordPress makes creating a great website quick and easy, and it’s a terrific option for most companies. But as your company is growing, consider making the move to a headless CMS part of your long-term vision; you will be more flexible, prepared for new devices, and have no worries to scale for high traffic. Understanding your options in content management can help you deliver the experience you want, and the one your customers expect.

PeakActivity can help bring your company to the next level by considering all your options in content management. By working with those in the know and using the right tools, you can instantly access new tools to help your website thrive.