Many sites have fancy seals like ‘HackerSafe’, ‘WebsSafeShield’,'BuyerShield’ and so on. Their issuers claim that those seals give confidance to customers and they are more likely to buy on site that has security seal. Some seal issuers claim boost of sales around 20%. The question is – is it true that store owner’s get more orders? How to determine that? If You’re store owner and Your store is using Zen Cart You can perform Your own A/B split test to find out if those seals improve sales.

First, do the math

Business is about income – if investment does not earn You more money than it costs You shouldn’t put money into it. Before You purchase security scanning service You should calculate if it is worth to pay $40-$80/month just to get the seal?

Let’s say that Your store has income of $200/month (hey, maybe You’re just starting?). In best case scenario You’ll get 20% boost in sales, which is $40/month. In not-so-optimistic scenario sales will improve by 10% – $20/month. You’re spending $40-$80/month for the seal, and it provides $20-$40/month, so You’re spending more than You earn. In that case You shouldn’t get the seal – it’s just not worth it.

Now, if You get $1000 or more out of Your store things start to look bit different. The seals earns $200/$100 per month in optimistic/not-so-optimistic scenario, so You earn more than You spend. You definitely should get the seal. Keep in mind however that You’re still unsure that the seal will provide more money than it costs. You need to perform A/B split test to know that.

The Test

I’ll show You exactly how to perform such A/B split test using Randomized Tests. You’ll need Randomized Tests contribution which allows You to perform A/B split tests on Your site. Randomized Tests requires also User Tracking Interface contribution which You can find here.

I will be using default Zen Cart template to insert the seal, but steps should be very similar with Your custom template. I also decided to use WebSafeShield seal in the test.

Step 1 – create the groupset

I want perform A/B split test with two test groups:

• Control Group – visitors in this group will see old version of the page without the seal.
• Test Group – visitors in this group will see changed header with the seal displayed.

Before I create the test I need first create set of visitor groups – groupset. To do this I go to Admin -> Tools -> [RT] Groupsets and I create ‘Standard Groupset’.

I then click twice on it and create two groups:

• Control Group (name: ‘Control Group’, id: ‘cg’, control group: checked)

• Test Group (name: ‘Test Group’, id: ‘tg’, control group: unchecked)

Step 2 – prepare the template

I want the seal to be clearly visible to visitors so I’m going to insert it into header area. With this in mind I change the template file (includes/templates/template_default/common/tpl_header.php) around line 77 from:

to:

Step 3 – create the test

Now You just need to create the test. Go To Admin->Tools->[RT] Experiments and use form on the bottom to create new experiment:

• Name – enter ‘Seal Test’
• ID – enter ’seal_test’
• Last From – enter current server date, e.g.: 2010-05-21 19:00:00
• Last To – enter test end date – I recommend entering date far in the future as You can always end the test earlier. Enter 2011-01-01 00:00:00
• GroupSet – choose ‘Standard’
• Impression Detection Function – leave as it is (‘default_rt_idf’)
• Click ‘Create’

Creation of the experiment

View of test stats just after the test has been created

Step 4 – verify that the test is running

Go to Your store and check that the test is running. Make sure that ‘Impressions’ counter in Experiment Statistics has been increased by 1 on Your visit (this counter is increased ONLY ONCE per visitor). Visit Your store from few computers or smartphones – You should be able to see modified version of the header on some (unfortunately Randomized Tests does not provide yet tool to store owners to switch test group – You have to test from several computers or clean cookies to be moved to different test group).

Header of Control Group

Header of Test Group

Step 5 – wait for the results

Wait until the test will provide statistically significant results. Result is statistically significant if some group is clear winner – it has 95% or more chance to beat control group and has been fed with enough data. If test group has 5% or less chance to beat control group You may consider control group as clear winner. As the rule of thumb let the test run until:

• One of groups get 95% chance to be clear winner
• and there has been at least 100 impressions and 50 order conversions in each group.

Step 6 – do the math again

You finished Your test and have the clear winner. If the clear winner is control group then it means that the seal does more harm than good and Your customers are less willing to buy when the seal is displayed (it would be strange though – maybe HTML code is not properly formatted and it breaks design of Your store?)

If test group (the one to which seal was displayed) is the winner You should check how much more money it earned. If ‘Total Orders Value’ rose by %10 and You earn $1000/month then the seal earns: 0.1 * $1000 = $100/month, so it’s a good idea to keep it.

If You decide to perform the test please post Your results in comments – I’m curious how much such seals boost sales in real life.

No tags