Tuning your website with A/B split testing (and a book update)
Thu, Oct 13 2005
One of the most powerful things about running a website is that everything is measurable. Every page view, file download, inquiry, and order can be recorded and reported in meticulous detail by your server.
Because everything is measurable we don't have to just sit back and hope that the way our sites are set up is the best they could be, or make changes such as updating content or the interface design blindly without knowing if we made things better or worse. You can measure critical performance metrics of your site to obtain a baseline reference, make changes, and then measure again. The numbers will tell you if you did the right thing or not.
Let me explain.
Imagine you have a website that sells items online, and you want to see if putting testimonials on your home page would increase your sales rate. You could just "try it and see" but without before-and-after figures you'd be working entirely on gut instinct.
A better approach is to determine your current sales conversion rate by dividing the number of visitors to your site each month by your number of sales. Imagine your site had 1643 visitors in one month, and you made 54 sales: that's a 3.2% conversion rate. Next you make a change to your site by adding some testimonials on the home page, and measure again: next month you had 2093 visitors and made 86 sales for a 4.1% conversion rate. That's a nice little change that netted you an extra 19 sales over what you would have made at the previous conversion rate.
But did it really? The figures look good and we can congratulate ourselves on measuring the results, but was there another factor involved? Maybe a good review of your product came out that month, and the increase in both visitors and conversion rate came about because more people arrived at your site predisposed to buy after reading the review. Maybe your change to the home page had no effect at all, or actually had the opposite effect but was offset by other factors.
Hmmm, now it's getting tricky!
We can solve that sort of problem using a technique known as "A/B split testing", which involves randomly serving up two slightly different versions of your site to different visitors and seeing which one works better. In split testing the objective is to isolate a single change such as the addition of testimonials to the home page and see what the result of that change is.
So in our example above what we'd do is prepare a modified version of the home page and have it served up in place of the normal version 50% of the time. Then we would track the conversion rate for both versions, and because both versions are being tracked simultaneously we don't have to worry about effects like traffic spikes or unusual events. At any point in time half the site visitors are seeing one version, and half are seeing the other. That makes comparisons between versions much more valid than if you ran one home page one month and a different home page the next.
A/B split testing is an extremely useful technique that in my opinion is underutilised at present. If you're making changes to your site maybe you should think about giving it a go.
On another topic, I'm about to start work on the second edition of my book "How To Build A Website And Stay Sane" and I'd love to hear your opinion on what issues need more detailed coverage. The first edition has been used by thousands upon thousands of people all over the world as a roadmap for their website (re)development projects and I've received a lot of positive feedback, but there are a couple of topics in it that I should probably cover in more depth. I've got my own ideas about what they are, but I want to hear your opinion! Make sure you let me know ASAP and I'll do my best to accommodate you.