A/B Testing on Black Friday

, Posted by Chris Wallis in Blog, Chris's posts, Conversion rate optimisation

Chris Wallis, Optimisation Consultant at Mezzo Labs, explores the benefits and risks of testing during peak promotional periods.

Black Friday is fast approaching, bringing with it fantastic opportunities for vendors great and small to test and optimise online. But it’s not all plain sailing…

The frenzied shopper is prone to panic-buying which could skew your results and provide insights not applicable during quieter times of the year. On the flip side, seeing how users react to a deadline, and how sensitive they are to it, can help you plan better for the next holiday season.  Does a possible drop in sales outweigh the potential for success?

A risky strategy for CRO

Let’s get down to it: What opportunities does this spike in traffic offer the conversion rate optimiser? There are two schools of thought on A/B testing during peak periods… Some see it as an opportunity to squeeze out more sales. Others would avoid it – it’s too important a trading period to risk mistakes. So, who’s right and who’s wrong?

Pro:  Higher visitors + increased conversion = even higher sales

Provided your baseline conversion rate doesn’t dip too much over the Black Friday period, more visitors mean more sales.

Why? Let’s say you run a test and it gives you a healthy 5% boost for your conversion rate. A 5% increase in a larger number of visitors is going to give you more than a 5% increase of a smaller number, right? Any improvement will be a greater improvement.

Traffic spikes like this don’t last forever, so make the most of it, but only with tests you have great confidence in. Get it wrong and you’ll be missing sales. Hedge against it by only showing the variant to, say, 20% of visitors.

Accurate, confident results

It can often be difficult to reach a reliable conclusion for a test if the numbers behind it are small. It could take weeks or even months for conversion rate uplifts to stabilise to something you can trust.

Fortunately, the accuracy of results improves with more data in your sample. Maximise test data by running tests when there’s a lot of activity on-site.

Multivariate tests need even higher volumes. So Black Friday might be a chance for that MVT you’ve been wanting to run.

More revenue generated

Another option could be to apply a bandit algorithm to an A/B test of your banner. As well as helping you find the best performer, it will help generate the most revenue during the testing period. It works by changing the weighting of the test from 50:50 to one that favours the winning variant to maximise the number of conversions.

Gain urgency insights

A/B testing is also a great way to measure customer reactions to special offers. Early season shoppers may be less time-sensitive than last-minute shoppers, where ‘finding the right gift’ might be the most effective message. For later season shoppers, the number of products left in stock and delivery timelines may be more relevant. Use stock counts, or offer expedited shipping options and make gift options more prominent to improve conversions.

Con: The risk of getting it wrong

If you test a variant that performs badly, the negative effects will hurt most over Black Friday. So carefully consider the possibility that the test variables won’t work. How radical is the change? And how could it go wrong?

Check how quickly your testing tool will report on performance, especially if your site receives a lot of traffic. Some tools have a lag of several hours before data appears, by which time it might be too late.

Read up on the statistical model that your tool uses. Some will find winners later than others to reduce the risk of declaring false positives.

Frankly, we prefer this approach, as such models are going to provide more truthful final figures. The issue is that it could mean a longer test duration, and could potentially leave you with fewer reported successes. Will your promotional period last long enough to get you a significant result? Ask your tool vendor for their test duration calculator.

Server load slows your website

During events like Black Friday, testing platforms can be strained by the volume of server calls to their service. SAS tools like Optimizely and Adobe Target support hundreds, if not thousands, of tests on more than just your site. They will have a dizzyingly large number of test pages to render. This causes a huge strain on their infrastructures, potentially resulting in increased latency.

Top vendors have built-in timeouts to safe-guard you against extended response times. If a response is not received within a set timeframe (a matter of milliseconds), the site will just load the default experience. However, there is a risk that this doesn’t happen. It might be worth asking what measures your vendor has in place to ensure visitors aren’t impacted.

Final Thoughts

Recycle the results

I’ve seen a lot of tests where businesses frantically try to promote a seasonal sale in the best way possible. Image A? Or image B?

That’s all well and good if you reach a conclusion quickly, and can apply the winner for as long as possible during the promotion’s lifetime. However, if you don’t get a strong winner, the test might be a waste of time and resources.

Usually, you can learn something from a test even if you don’t get a winner. But when it’s specific to a unique event, there’s nothing to carry forward.

Always make sure the learnings are transferrable in some way.

Watch your audience types

Events like Black Friday see businesses inundated with deal-hunters, many that have never visited your site before. They know what they want, but why should they buy it from you?

Make sure your offer and USPs are clear. This is a good opportunity to test how you should be promoting your brand values.

Test discount levels

Test price elasticity, too. How much should you cut your prices to get a sale? Cheaper stock may drive more sales, but will it erode net profits? The trade-off between transaction volumes and AOV is different for every brand and well worth testing to find the optimal balance. Insights gained are perfect for carrying over to next year, when it all starts again.

The verdict

On balance, it’s worth running tests during the Black Friday peak. Provided you manage the risks, there are insights to be gained, sales to be made and new customers to be won. If you don’t feel confident testing during the holiday period, there are other ways you can learn about your users, such as running focus groups, using website feedback tools and recording customer questions to see how you can better answer their needs.

Whatever you decide, good luck. We’re here if you need us.  

Contact us

Chris Wallis, is an Optimisation Consultant at Mezzo Labs and an expert in the AB testing journey, from initial strategy and build, through to in-depth statistical analysis.