Split Test Best Practices
The real key to successful testing is traffic volume.
The lower the traffic numbers, the larger the difference in conversion rate needs to be to decide on a winner. Statistical significance depends on numerous factors but the most important is driving enough traffic to the test to have confidence that the test showing the highest conversion is genuinely a ‘winner’
Quality of traffic is next.
If you direct thousands of untargeted visitors to a test (visitors who are unlikely to ever become subscribers or customers) then you’ll see corrupted results. Traffic should always be targeted, visitors who will be interested in what you sell or offer.
Finally, Test Smart:
We recommend you limit each test to a single element on your pages, determine a winner and then create additional tests for other elements. (Elements are things like your headlines, the order buttons, page images, colors, call to actions etc.)
The more variations you have in a single test the longer it will take (or the more traffic you will need) to determine what’s working.