Randomized tests version 1.3 is here (download). This release brings quite few new features that should be very helpful. I added 95% trust intervals for each statistic. There’s also probability that given experiment will beat control group. You can now also decide when impression occurs. Read below for more.
Installation
Refer to INSTALL.txt in .zip file.
If You already have RT installed refer to INSTALL.txt file for UPGRADE instructions.
Trust intervals
Randomized Tests provides few estimates:
- order conversion rate
- visitor to customer conversion rate
- total items bought
- total order value
You have to remember that RT presents only estimated values – for example after tracking ten orders RT might calculate that order conversion rate is 0.9%, but You’re almost sure (because You know that from earlier experiments) that Your conversion rate is 1.1%. Unfortunately RT can’t do much better since it hasn’t collected enough data yet. In new version RT will however present also 95% chance intervals for each estimated value. So now instead of conversion rate 0.9% You’ll see 0.9% +/- 0.2% which means that there’s 95% chance that true conversion rate is between 1.1% and 0.7%.
Chance to beat Control Group
Previous versions of RT leaved to You decision if changes You made were actually useful. In version 1.3 RT will give You probability that those changes outpeform original version of Your Zen Cart store. RT displays now ‘Chance to Beat Original’ value. This value should be around 95% or 5% before You decide that new version perform better/worse then original version of Your store.
Custom impression detection functions
This new feature lets You decide if and when to include particular visitor in experiment. You might want for example test performance of Your new customer registration form and include in experiment only those visitors who actually visited customer registration form. Thanks to that You’ll get more accurate results very quickly.
There will be separate post about this new feature – so stay tuned 
Download
You can download latests release of RT here:
www.data-diggers.com/contribs/rt/downloads/rt.zip
2 Comments for Randomized Tests version 1.3
Raine | February 1, 2010 at 17:33
Author comment by admin | February 10, 2010 at 20:44
If it’s possible I’ll do it. But from I know it isn’t as the don’t provide any API.
Is there a chance that we will see some kind of integration with google a/b split test?