Google Experiments – Are they accurate?

Last year Google discontinued their Website Optimizer service in favor of “Experiments” which resides within Google Analytics under the content tab. Both options were intended for you to compare different website layouts to see which converted best.

The fact that Google offers experiment services like this free of charge is incredible. As with most Google products I assumed that Experiments would work correctly, but after some changes where the analytics didn’t match up with experiment results I decided to do a test.

The Experiment

I created an Experiment with one page (Original) versus another page which was identical in every way (named “Same as original” as I wasn’t feeling creative that day).

The experiment ran for 34 days and saw 21,693 visits with 1360 conversions so I’m confident that this is a reasonably sized sample and we can draw conclusions from it… or can we??

The Results (caution they’re alarming)

Right off the bat page “Same as original” took off and was surprisingly performing far better than the Original. You’ll see in the experiment that as time went on a larger portion of traffic was directed towards the supposedly superior “Same as original” since it seemed to be converting better.

In reality though “Same as original” should not have been performing better since both pages in the test are identical.

In the end “Same as original” had a 6.42% conversion rate which was an impressive 21% better than Original which had a 5.28% conversion rate. Google was 94.4% confident that it would out perform the original so they didn’t feel it was completely conclusive but these numbers sure would make me think they were.

Here’s how it looked:

Google Analytics Experiments Test

Final Thoughts…

It’s great that Google offers free tools but these results are scary since these numbers didn’t reflect reality and in online marketing changing something a percent or two could mean thousands of dollars or lost revenue.

Sure some of you may say I should run this test again, but if you’re like 99% of the websites out there then you’d have to run the test for a year just to see impression numbers like this, yet this tool is marketed as something that everyone can use to help improve their site.

Anyone else with similar results?