Quote:
Originally Posted by Robbie
Seems like at the end you still have to make a critical judgement yourself though. What if an inordinate amount of traffic comprised of adults with credit cards who will actually buy a membership went to "A" just coincidentally?
And at the same time...also coincidentally, a big percentage of guys without credit cards or who won't buy porn went to "B".
In that scenario, you could potentially have a much better "B" join page that would perform better than "A", but you wouldn't know it.
Of course I'm assuming that you run the test for a while and not just one day.
But even then...if one of the pages performs better than the other one...you might be losing members who went to the underperforming one.
I guess you have no choice but to do it though.  |
Hi Robbie, I really think you are over thinking this. Any kind of testing of this sort isn't ever going to be 100% but the aim here is to give yourself the best possible chance you can. By not testing you run similar, perhaps worse, risks simply because you don't know. What you don't know, when it concerns conversions, is always worse than what you do know with some margin of error built-in.
These tests, even if not 100% accurate, don't use small sample rates. For instance, the last google experiment I ran on a 3 different versions of a page lasted 30 days and 120K users before it decided a winning page for me. I think people with more traffic would get there quicker but that is typical for any experiments I have run in the past.