Quote:
Originally Posted by Paul Markham
Very true.
If one month the traffic that comes stops signing up and the next month the traffic start signing up again. For no reason what so ever it will make the whole experiment rendered useless.
What's the odds of that then?
|
The odds of completely on and off? Miniscule.
The odds of it being a different time of the month (pay cycles have a huge influence on your stats, most people get paid near the 1st and 15th of the months), different season, closer to the holidays, etc etc etc... are huge.
This test will almost certainly show one way or the other, but you'll never know if it was accurate or not.
There is also the problem of statistical certainty. You need to exceed the standard deviation of your signup ratios in this test, something that isn't likely to be monitored correctly without the right software.
</shameful plug for my software>.
We talked about my multivariate testing suite before... hit me up if this is still something you'd like to pursue. I'll guarantee a 20% increase in signup ratios across the board; money back if it doesn't work.