Category Archive:
Posted by admin on May 18, 2010 at 13:36
Randomized Tests version 1.3 introduces new feature – custom impression functions. It allows You to decide who and under what conditions participates in A/B split test. In this post I would like to show You how to create custom impression detection function and how to use it to speed up experiments.
The need for custom impression detection functions
Let’s assume that You want to change Your registration page. You hope that new version will ease process of registration and more customers will register/order. You want to include in experiment only those visitors that actually viewed ‘Register’ page and remove from experiment those visitors that never intended to register / make purchase and never visited ‘Register’ page. Why? Because You can get more accurate results more quickly. Let me show it to You on example.
Example
Let’s assume that Your site gets 1000 visits daily (excluding bots). Of those 1000 visitors only 100 intend to buy anything ( other visitors are just surfing, comparing prices or doing anything else – they are not customers). Those 100 visitors get to ‘Register’ page where You already set up A/B split test with two versions of the page. You know – from Google Analytics or anywhere else – that old version of the page (version A or control group) has 20% conversion rate. You expect new version of the page to have 40% conversion rate. How quickly will You know that the new page is better? Here’s comparison of default IDF vs custom IDF:
- Default impression detection function (all visitors are included in the experiment)
| Group |
Visitors |
Impressions |
Conversions |
Conv. Rate |
Duration |
| Group #A |
500 |
500 |
10 |
2% |
6.41 days |
| Group #B |
500 |
500 |
20 |
4% |
- Custom impression detection function (impression occurs only when visitor views ‘Register’ page for the first time)
| Group |
Visitors |
Impressions |
Conversions |
Conv. Rate |
Duration |
| Group #A |
500 |
50 |
10 |
20% |
5.66 days |
| Group #B |
500 |
50 |
20 |
40% |
As You can see from this simple example there’s 15% decrease in duration of experiment. In other scenarios improvement may be even better.
How to create IDF (Impression Detection Function)
Randomized Tests can make use of any function that:
- is located in ‘includes/functions/extra_functions’ directory
- ends with ‘_rt_idf’ string (e.g. my_custom_rt_idf())
- takes five arguments in following order:
- $experimentRowID – ID of row in rt_experiments table representing current experiment
- $experimentID – ID of current experiment You created (e.g. ‘login_test’)
- $groupRowID – ID of row in rt_experiments_groups table representing group to which customer has been assigned
- $groupID – ID above group (e.g. ‘control_group’, ‘test_group’)
- $newlyAssigned – TRUE if visitor has just been assigned to group (e.g. it’s his first visit to Your store)
- returns:
- TRUE – if impression occured
- FALSE – otherwise
Ok, so let’s create such function and upload it to /includes/functions/extra_functions:
function visited_register_page_rt_idf($e_row_id, $eid, $g_row_id, $gid, $new) {
global $uti;
global $current_page_base;</p>
if($current_page_base == ‘create_account’) {
$flag = $uti->get(‘visited_create_account_page’);
if($flag === FALSE || $flag == ‘FALSE’) {
$uti->set(‘visited_create_account_page’, ‘TRUE’);
return(TRUE);
}
}
return(FALSE);
}
First, function checks if currently visited page is ‘create_account’ page – We want impression to occur only on registration form page (which in Zen Cart is ‘create_account’ page). Impression should occur only once per visitor so function checks using UTI if visitor already viewed ‘create_account’ page. If he didn’t function does two things:
- saves information that visitor viewed ‘create_account’ page in UTI
- return TRUE – impression occured
If visitor already visited ‘create_account’ page impression does not occur and function returns FALSE.
Now upload the function to includes/functions/extra_functions and create experiment that uses it to detect impression.
Modify ‘Create Account’ page
Let’s modify ‘Create Account’ page. We just want to test two versions of ‘Create Account’ page so I used ‘Standard Groupset’ (Admin->Tools->[RT] Groupsets) as groupset. Standard Groupset has two groups: control group (id ‘cg’) and test group (id ‘tg’).
if(!isset($exp) || $exp->groupID == ‘cg’) {
// control group
// display old ‘Create Account’ page
} else {
// test group
// display new version of ‘Create Account’ page
}
If visitor has been assigned to control group (‘cg’) old version of page is displayed. In other case new ‘Create Account’ page is used.
How to use IDF in experiment
Go to Admin->Tools-> [RT] Experiments page. Create new experiment as usually. In ‘Impression Detection Function‘ box paste name of Your impression detection function: ‘visited_create_account_page_rt_idf’ (without braces). Click on ‘Create’. That’s it!
Test that Your IDF works
Visit Your site and check that impression only occurs on first visit to ‘Create Account’ page.
That’s it. If anything isn’t clear please post comment.
Continue Reading
Posted by admin on November 23, 2009 at 11:05
Randomized tests version 1.3 is here (download). This release brings quite few new features that should be very helpful. I added 95% trust intervals for each statistic. There’s also probability that given experiment will beat control group. You can now also decide when impression occurs. Read below for more.
Installation
Refer to INSTALL.txt in .zip file.
If You already have RT installed refer to INSTALL.txt file for UPGRADE instructions.
Trust intervals
Randomized Tests provides few estimates:
- order conversion rate
- visitor to customer conversion rate
- total items bought
- total order value
You have to remember that RT presents only estimated values – for example after tracking ten orders RT might calculate that order conversion rate is 0.9%, but You’re almost sure (because You know that from earlier experiments) that Your conversion rate is 1.1%. Unfortunately RT can’t do much better since it hasn’t collected enough data yet. In new version RT will however present also 95% chance intervals for each estimated value. So now instead of conversion rate 0.9% You’ll see 0.9% +/- 0.2% which means that there’s 95% chance that true conversion rate is between 1.1% and 0.7%.
Chance to beat Control Group
Previous versions of RT leaved to You decision if changes You made were actually useful. In version 1.3 RT will give You probability that those changes outpeform original version of Your Zen Cart store. RT displays now ‘Chance to Beat Original’ value. This value should be around 95% or 5% before You decide that new version perform better/worse then original version of Your store.
Custom impression detection functions
This new feature lets You decide if and when to include particular visitor in experiment. You might want for example test performance of Your new customer registration form and include in experiment only those visitors who actually visited customer registration form. Thanks to that You’ll get more accurate results very quickly.
There will be separate post about this new feature – so stay tuned 
Download
You can download latests release of RT here:
www.data-diggers.com/contribs/rt/downloads/rt.zip
Continue Reading
Posted by admin on October 22, 2009 at 21:14
Yeah, I’m not dead. I don’t have much time but I still work on my modules. I spent most time on Randomized Tests module – sorry to all people counting on new Query Cache version . In upcoming release of Randomized Tests You’ll be able to:
- decide whether or not impression occurred (and when it happened). For example, You want to count in experiment only visitors that viewed ‘login’ page. Now You can
- get more accurate results. Experiment stats view in Admin area will give You probability that given test group will beat control group. All stats will be also presented with 95% probability range. For example visitor/customer conversion ratio will be presented as: 1% +/- 0.02% which mean that conversion ratio is in [0.98%, 1.02%] with 95% probability.
- maybe more… for example support for Your own statistics (like ‘search form usage stats’, ‘contact us page stats’ etc).
Would You like to see more features in next version? Let me know through comments / mail.
Sorry, multivariate testing will probably not be included in upcoming version. I’ll add support for it most probably in next next version.
Continue Reading
Posted by admin on July 20, 2009 at 13:08
New version of Randomized Tests for Zen Cart is available. New version introduces few bug fixes and one very important feature – orders filters and customer filter. Order/Customer filters can be used to dynamically remove (not permanently) from statistics certain customers/orders and recalculate statistics without them. Why is it important?
Let’s assume that You have store with many customers. You use Randomized Tests to check how new layout affects sales. Unfortunately after You run Your experiment for few days You notice that one group is biased because it includes one or two top buyers. Those two buyers are different from regular customer – they buy a lot of items, place orders every few days and probably aren’t affected by new layout – they already know Your store and don’t care about it as long as they can place orders.
Group containing these customers is biased since it includes data outliers. You probably would like to remove them from statistics. New version of Randomized Tests can do that using filter functions. Currently there are two types of filter functions:
- order filter functions – executed for each order in group – allows removal certain orders from statistics.
- customer filter functions – executed for each customer in group. Orders from rejected customers will not included in statistics.
Randomized Tests V1.2 contain two sample filter functions:
- ddigers_2std_customer_total_rt_cff() – removes top buyers (formally – customers with total spent in 2+ standard deviation from average) from statistics.
- ddigers_2std_order_total_rt_off() – removes super big orders from statistics.
To check how they’re implemented view admin/includes/functions/extra_functions/randomized_tests.php
You can implement Your own functions by following below pattern.
Custom Customer Filter Functions
Randomized Tests will recognize any function with name ending with “_rt_cff” as customer filter function and allow You to use it. For example following function will be recognized as filter:
function customer_filter_rt_cff($customer_id) {
if($cid == 12345) return(FALSE);
else return(TRUE);
}
Randomized Tests passes to customer filter functions single attribute – id of customer in database. If function returns TRUE customer is accepted in calculation of statistics, and is rejected otherwise.
Order Filter Functions
Order filter functions are similar to customer filter functions – any function with name ending with “_rt_off” will be recognized as order filter function. Example filter function:
function order_filter_rt_off($eid, $gid, $cid, $oid, $itemsBought, $ordersTotal) {
if($orderTotal > 1000.0) return(FALSE);
else return(TRUE);
}
Randomized Tests passes few more parameters to order filter functions:
- $eid – experiment row id in database
- $gid – group row id in database
- $cid – customer id in database
- $oid – order id in database
- $itemsBought – number of items in order
- $totalValue – total value of order
As with customer filter functions order filter functions should return TRUE to accept order in statistics and FALSE to reject it.
Making Filter Functions Visible to Randomized Tests Module
Let’s assume that You’ve placed Your custom filter functions in admin/includes/functions/extra_functions/my_custom_filters.php file. The best way to make them visible to RandomizedTests is to include the file through autoloaders interface. Create config.filters.php file in admin/includes/auto_loaders/ and paste there:
$autoLoadConfig[180
][] = array(‘autoType’=>
;‘require’,
‘loadFile’=>
; DIR_FS_CATALOG
. DIR_WS_FUNCTIONS
. ‘extra_functions/my_custom_filters.php’);
That’s it – Randomized Tests module will now be able to use Your filters.
Screenshot
Here’s screenshot of experiment with applied filters:
Download
You can download current version of Randomized Tests here.
Continue Reading
Posted by admin on July 14, 2009 at 14:38
This module allows You to perform experiments on Your website. You can test for example how Your new ‘Add to Cart’ button performs compared to old one.
I don’t have much time to write long post, so I’ll cut it to most important things. You can download the module here. This module requires UTI to work properly (download it here ). Installation instructions are in .zip file. Here’s how You can use it.
To create experiment follow these steps:
- Go to Admin->Tools->[RT] Groupsets.
- Create groupset (it’s set of groups). Call it ‘Basic Test Groupset’.
- Click on Basic Test Groupset – You will be forwarded to page where You can edit it.
- Each groupset must have EXACTLY one control group. I usually call it ‘Control Group’. Create two groups (you can create as many groups in groupset as You want, but for simplicity We use 2 now):
- Control Group – with id ‘cg’ (as ControlGroup). Check ‘control group’ checkbox
- Test Group – with id ‘tg’. DO NOT check ‘control group’ checkbox.
- Go to Admin->Tools->[RT] Experiments.
- Create Experiment ‘Exp #1′ using groupset ‘Basic Test Groupset’.
Click on Exp #1 – You will be forwarded to its stats:
- Impressions - in this version of Randomized Tests its simply number of visitors in each group
- Customers Registered – (visitor to customer conversion rate) / (change in regard to control group) / (total count of newly registered customers in this group)
- Orders Placed – (visitor to order conversion rate) / as above / (total number of orders visitors in this group placed)
- Items Bought – (avg. size of order) / as above / (total number of items bought)
- Orders Value – (avg. order value) / as above / (total orders value)
Ok, You’ve created experiment, now You have to create two versions of Your page. Let’s assume that You want to change ‘Add to Cart’ button on product_info page. Edit includes/templates/your_template/templates/tpl_product_info_display.php.Find code responsible for displaying ‘Add to Cart’ button – in my case it’s:
Change it to:
if($exp->groupID == ‘tg’) {
// visitor has been assigned to Test Group
echo $display_my_new_button;
} else {
// user is in Control Group – We don’t change anything
echo $display_button;
}
$exp is global variable (it’s instance of ExperimentManager defined in includes/classes/randomized_tests.php) and holds basic information about experiment and group to which has been assigned current visitor
$exp->groupID is ‘id‘ of group that You assigned in step 4. You can use it to detect in which group is current visitor and perform custom action.
You’re done. Wait until experiment becomes statistically significant ( >2000 visitors in each group, many orders ) and check results.
Continue Reading
