A/B Split Testing Magento with Google Analytics (CRO)

There are a number of different ways to split-test websites and webpages to see which design or layout converts better. It’s all about CRO – a bit of a trend in web design at the moment (along with UX). CRO stands for conversion rate optimisation – but if you’re reading this post, you probably already know that.

This particular post is a “how-to” for a new (and somewhat experimental) way of split-testing your Magento website using Google Analytics.

There are a lot of tools out there already that allow you to test individual pages – and then track events, ecommerce conversions etc from those pages. This method however is different, it’s more of an old-fashioned method but one which allows you to change your entire theme per visit – not just the styling of a landing page.

If you’re after the individual page kind of testing I’d recommend you take a look at Google Experiments – or alternatively (and a far better tool) Optimizely.

This post is split into the following sections:


  • The Theory Skip
  • Creating your Analytics Accounts Skip
  • Installing the 1.7+ Module Skip
  • Configuring the 1.7+ Module Skip

The Code

  • Where to find the code Skip
  • The main coding elements Skip


The Theory

The theory is simple – you want to make changes to your Magento store – you want to track how those changes affect customer behaviour on your website – however you want to compare these changes against your current theme. The only way to do this effectively is to serve 50% of your customers with the new theme – and keep the other 50% of your site traffic landing on your current theme. This way you don’t have to worry about timeframes being different (for example seasonal / weather-related purchases) – as the split testing all happens at the same time.

So when we are split testing it’s important that the following critical factors are covered:

  • Ability to serve exactly 50% of site traffic to the new theme
  • Ability to analyse and contrast the new 50% compared to the old 50%
  • Keep our normal “master” analytics account untouched (so as not to disrupt our data)

Here is how we are suggesting we split-test Magento using Google Analytics:

  • 3x Google Analytics Accounts
  • 1x master Google Analytics account to track ALL visitors (this should be your current account)
  • 1x “A” Google Analytics account (only shows for the current theme)
  • 1x “B” Google Analytics account (only shows for the new theme)

With these accounts set up we should be able to:

  • Track all visitors as normal in a MASTER analytics accout
  • Track 50% of the overall visitors movements / purchases within our new theme (Account A)
  • Track 50% of the overall visitors movements / purchases within our current theme (Account B)
  • Easily Draw comparisons between Account A and Account B using all of Google Analytics tools

Here’s a little diagram to make it a little clearer (possibly):

How a visitors is given the A or B theme, different data in different Google Analytics Accounts
How a visitors is given the A or B theme, different data in different Google Analytics Accounts

You can then easily measure things like conversion rate %, total revenue etc.

Creating your different Google analytics accounts

So, we now know the theory, how do we put it into practice? The first thing to do is to create your Google Analytics Accounts. I’m assuming that you already have a master google analytics account set up – if not simply create one by going to your account list and clicking Add New Account.

Within this new account area you’ll be asked to complete your new PROPERTY. For the time being please use “Classic Analytics” – I’m not 100% sure that Magento has the ability yet to use the new version dynamically. Proceed to enter all your details then save.

The next thing to do is to create two more identical PROPERTIES within this account. You can navigate back to your account list by clicking on its name in the breadcrumb. Add your properties (name them something like Old Theme, Current Theme) – again making sure that they are using “Classic Analytics” mode.

You should end up with something that looks like this:

You should end up with 3 properties for your account. Each should have pretty much the same Property ID (except for the last digit)
You should end up with 3 properties for your account. Each should have pretty much the same Property ID (except for the last digit)

Make note of these property ID’s (tracking codes) as we’ll be using them in our Magento module.

Installing the Magento Module (1.7+)

Unfortunately due to the way Magento displays it’s analytics implementation through 1.4/1.5/1.6 (not really using the ga.phtml) this module is only applicable in Community Edition 1.7+. I’ll demonstrate the actual code itself later in this post for those who want to use this module as a starting point for developing a compatible module for earlier versions of Magento.

You can get the Magento AB Testing module here.

Once installed please make sure that you clear your cache and logout / back in again!

Configuring the module

Now that the module is installed, make sure that your current analytics account settings have been setup correctly.

Navigate to System > Configuration > Google API (on the left) and make sure it’s enabled and your MASTER account GA code is in there.

Ensure your master google analytics settings are setup correctly
Ensure your master google analytics settings are setup correctly

Once that is completed your analytics code should start showing up in your tag.

The last thing to do is to modify your AB testing settings. Navigate to System > Configuration > AB Testing and then use the simple form to select your current theme (in this case default) – paste in your current theme analytics code, then select your new theme and paste in it’s property ID. Set the module as Enabled and you’re good to go.

Select your current theme and your new theme - pop in the property ID's for each.
Select your current theme and your new theme – pop in the property ID’s for each.

Now that’s done we should give it a test.

Testing the module

Open two browsers – with one visit your website – you should be given the new theme or the old theme, try again in your other browser and you should be given the alternative theme (as long as you’re the next visitor). This alternation will happen every time a visitor visits the website – however, because their choice is saved in a cookie, as they browse the website it will stay using the same theme.

To double-double check, let’s open the source code for each theme – we should see two sets of analytics codes – one for our MASTER account and one for our THEME X account.

In the head section we can see that we're bring out two versions of the analytics code
In the head section we can see that we’re bring out two versions of the analytics code

Again, if we were to see the other theme the other alternative tracking code should appear – but the master one should remain constant.

The Code

Where to find the code

If you want to edit this module yourself you can download the code from our Github Page.

The main coding elements

The main coding elements to this module revolve around the Google Analytics template file and also a few observers to set the theme.

Modifying ga.phtml

Within the module we modified the ga.phtml and remapped Magento to use our template (using our layout.xml). Within this file (frontend/base/default/creare/abtesting/ga.phtml) we’re doing a few bits:

  • Duplicating the whole script tag
  • Wrapping the duplicated script tag in our own functions (to check for enable/disable etc)
  • Setting _getOrdersTrackingCode to a variable so we can use it elsewhere (for some reason, by default, this method gets cleared so you cannot use it more than once – resulting in no ecommerce analytics!)

That really is all there is to the new ga.phtml file. As you can see it still uses the classic analytics method – for those developers out there who wish to attempt to use the new GA tracking code – this is the best place to start editing.

Extending Mage_Core_Model_Design_Package

The next step for us was to hi-jack where the themes are being set within Magento. We found that by extending the class Mage_Core_Model_Design_Package we could override the public function getTheme (you can find our override in app > local > Creare > Abtesting > Model > Package.php).

Within this function we’re performing the following.

  • Fetch our A/B cookie
  • Fetch our temporary session A/B variable
  • We then check to see if our module is active, we then set our class property _theme to the theme we wish to serve – based on whether the visitor has been given A or B
  • We then update a small config data element with the opposite theme – so next time someone comes onto the site they receive the different theme.
  • We’ve also thrown in a few session variables to make sure that we’re not setting cookies all over the place (remember – cookies are set – but are not read until the page is refreshed – so we temporarily use session variables so we’re not doubling-up the code)
  • Finally we use a simple observer to clear our session variables.

The rest is simply creating an admin area where you can select your theme / package and input your GA code.

That’s about it, we’ve also added in a couple of methods in the helper whereby the default GA code is brought back on it’s own if we have neglected to miss out a few things in the configuration.

If I’ve missed anything out, or if you have any questions please leave your feedback below, on the Magento extension page or give me a tweet (kent_robert).

  • Chris

    Hi Rob, thanks very much for this explanation and your free module. We have set this up and tested it for a week. Unfortunately we can not get this to work at the moment. The first day, only site A was shown. The day after, only site B was shown. We thought for a minute that it’s not based on visitors, but on days. However, now only store B is shown and store A is never shown to anyone. Do you have any idea how we can fix this? I have some screenshots, maybe you can send me an email so I can reply the screenshots to you? Thanks in advance!

    • Hi Chris, Sure you can find me though the contact form on our website – I’ll pick it up and email you back. Essentially the script should save cookies on the user’s machine to “persist” the given theme for a certain period – so it would only present you with the alternative theme (if you’ve already been given a theme) when your cookie runs out. I believe the default cookie setting is set to 1 day at the moment. If you can send a quick email though the form I’ll contact you back and we can have a look at those screenshots.

  • Hi
    I have installed your extension and tried to use it on our Magento store only to discover that it breaks things down. We are using varhish for caching and when the AB testing is turned on the site does not work. Further debugging shows that your package tries to set up the cookie in some kind of a loop breaking up html document structure which varnish catches and reports an error shutting the store down. Do you by any chance have a fix for that?

    • Hi Tomasz, Unfortunately the Varnish cache issue is a known problem – we haven’t figured out a work-around for it yet but we’ll let you know if and when we do

      • I think it’s not varnish issue per say, it’s the fact that the getTheme() method is being called hundreds of times during execution and varnish is handling it the way it does. I think you should look into storing result of your code in session and performing actual check only if the session variable is not set. this way you’ll not only fix the cookie problem but also speed up greatly code execution. For now I have fixed it on my end in a way that the cookie is set only once no matter how often the method is called and it works great.

        • Hi Tomasz – great spot! Would you mind contributing to the github repository with that change? I think other users will greatly benefit from those tweaks

          • done

          • Tiago Reganha

            Has the plugin on the magento commerce been updated? I use Varnish too and I’m desperate to start A/B testing :/

            Thank you,

          • Dharmesh

            Configured everything… !! but there is sometime some user can’t able to login in frount end and backend.

            Any solutions for this?

          • Dharmesh

            Can you please share this change Tomasz OR Robert?

  • John Paul Medina

    I was about to write a theme switcher plugin for A/B testing when I stumbled upon yours. Great job by the way. So instead of reinventing the wheel I took yours for a spin. I do have some issues though. It doesn’t seem to switch when I am using a different namespace. It’s almost like the theme needs to be part of the same namespace. I can live with that so I moved my theme into the same namespace and it worked. It seems to be switching fine however there was a couple other issues I saw.

    I am using a theme URL function to pull specific images however they are coming back as not found but they do in fact exist. This problem only happens on my B theme.

    My A theme is the same as the Magento set theme but my B theme is the new theme within the same namespace. I literally duplicated my original and gave the background a solid red color to just test.

    When I inspect the missing images it seems to be pulling from /skin/frontend/default/default/ instead /skin/frontend/NAMESPACE/theme

    Is this a known issue? I will continue to look further into this on my end but decided to post the question in the meantime.

  • aftabnaveed

    Extension looks promising, but it doesn’t work with Lesti::FPC full page cache module, I am trying to figure out why, if some one already knows why, and share the solution that would be awesome.

    • aftabnaveed

      We made it work by applying the patch provided by Tomasz, thanks mate

  • sandesh

    Thanks for this gr8 extension. I tried it om Magento CE Its works properly only if ‘Configuration’ cache is disabled. Any idea why this happening?

  • Bruno Marques Caldeira

    Nice work, looks really promising. Just one question, does it work in the current Magento CE 1.9? Thanks

  • Dharmesh


    I setup everything but sometime there is problem with login on frount end and backend. some user can’t able to login.

    Did you updated that code which Tomasz made comment here?

  • Brian

    This is fantastic, having a problem on, I am getting a 502 bad gateway when the module is implemented. I tested the themes individually and they both work on magento. Any idea on how to fix this? Thanks.

  • Patricio Köhler

    I’m using Lesti::FPC and google SpeedPage mod. One way or another these caching issues are returning always the same test a or b until the cache is cleaned. Do you have any suggestion I could try before digging into the code? (I’m an experienced PHP developer but I have never coded things for magento).

  • Kristian Brødsgaard

    I’m looking at using your fine plugin for A/B testing, but it appears (at least in magento, if configuration files are cached (system->cache), the setAbTest value is never updated, and only shop A is loaded.

    Is this an issue with this version of Magento?