A/B testing could be a big sham

By April 16, 2015 Links, marketing No Comments

Following on from this article http://snip.ly/ThMt

Here’s a few questions to ask yourself to decide whether you should run an A/B test:

  • Do I have an important question? Will answering this question make an impact worth the effort of running a test?
  • What else could I be doing with my energy? Running experiments and being creative and visionary are two completely different brain modes. Are you distracting yourself from making valuable insights?
  • Can I run a large enough test? Plug your baseline conversion rate and “minimum detectable effect” this sample size calculator to see if you have a chance at even reaching statistical significance.
  • Do I really understand testing? Try running A/A tests to get a feel for how misleading “results” can be. If you still feel testing is worth it, read and understand some of the resources below to learn how to run a reliable test.

Also, keep in mind there are other ways for businesses to “test.” I really like the lets-try-this-and-see-how-it-goes-then-iterate test.


About Kingi

In 1997 Kingi made a creative start at an advertising agency, Saatchi & Saatchi. Then, for five years was involved in the video game business, firstly in distribution and then as a developer. In 2004 Kingi started Ignite Studios and has been freelancing on all types of digital projects.

Contact us to request a quote for your project Email Us Now