Articles in The 'A/B & Multivariate Testing' Tag


July 17 2008

This types of stuff happens eh-veh-ree-DAY!

by MoreVisibility

In my latest, most desperate of attempts at trying to make our loyal blog readers think I’m hip by using titles that come straight out of popular phrases in rap songs (which is in conjunction with my last attempt with a blog post entitled “Tryin’ to make a dollar outta fifteen cent!“), I’d like to give you an idea of what the typical day-to-day life is like here for me at MoreVisibility. Every time I describe what I do to friends, colleagues, co-workers and even some clients, I talk about how being in Web Analytics is like being a private investigator or a federal agent of the internet. You gather data, compile statistics, find clues, compile some more data, interview a couple of people, and solve the mystery! Then you typically have to present your findings to your boss(es) and your clients, and then talk about where to go from there.

Here’s an outline of a typical day for me (which is sort-of a false statement, because no two days are the same, so there really is no such thing as a “typical” day…but you get the idea).

Date: Wednesday, July 9, 2008, Boca Raton, FL, USA (Temp: 91°)

7:04 AM – I have just woken up, and I’m already thinking about what I’m going to be doing for that day. Do I have an Analytics presentation to give? Do I need to check the coding on a site before it launches? What accounts will I be doing some investigating on? Do I have enough laundry to last until the weekend?

7:57 AM – I arrive at my office, turn on my computer, and see a yellow sticky note on my monitor that reads “Joe – Please see me about [Client]’s Top Landing Pages.”

8:01 AM – While my computer is loading and my email is downloading, I catch my co-worker who explains that our client is concerned that the exits from their homepage is too high. I suggest evaluating the page’s Bounce Rate and maybe a quick Navigation Summary to get a better idea of what is really going on with their homepage. I also mention something about A/B testing with Google Website Optimizer.

8:02 AM – I log-in to my Google Reader account and catch-up with the 60+ Web Analytics and Search Marketing blogs that I subscribe to, while simultaneously responding to emails with questions and discussions from co-workers.

8:41 AM – I am finalizing my speech for an in-person Analytics Presentation to one of our clients, when Amber (Client Strategist) buzzes me and tells me her client added an email address to their Google Analytics account, but they cannot log-in. She tells me she knows what the reason is: “The Email address is not a Google Account yet! It needs to be a Google Account in order to log-in with that Email address into their GA Account.” I start smiling, because that’s exactly right.

9:15 AM – I meet with Shawn, our newest co-worker, and begin to review all of the great things that is Web Analytics. Of course, I have to throw in my private investigator / federal agent simile. I also explain that Javascript-based programs like Google Analytics are only able to collect data from users who have both Javascript and Cookies enabled on their browser of choice. If they don’t, Google Analytics simply cannot track those individuals.

10:30 AM – I am out of water, and I’m starting to get hungry. I think about all of the different possible ordering options, and think how cool it would be if some of our favorite local take-outs would have an online ordering option, and imagine what I would give for a large turkey & swiss right now.

10:35 AM – I start to open up a brand new Google Analytics account for a new client. I provide our client with the necessary tracking code to be placed on every single page of the website. I also explain the many different options available, such as SiteSearch, Ecommerce, Benchmarking, and Filters that can be utilized.

10:59 AM – I receive a phone call from another client who asks me to explain the difference between A/B Testing and MVT (Multivariate Testing). We throw around some ideas of what to test and experiment back and forth, and we agree to launch an experiment using Google Website Optimizer for their AdWords Campaign’s landing page.

11:33 AM – Okay I am REALLY hungry right now and I can’t imagine being able to last another 27 minutes without eating something!

11:34 AM – Marni (another Client Strategist) sends me an IM that reads “It’s working!!!” She is referring to the neat advanced filter that we wrote which added the name of the source and the visitor type in front of the transaction ID in this particular client’s Ecommerce Report section. This is great news, as I’m sure the client will be very happy to hear about this.

12:00 PM – I’m about to grab my sunglasses and walk across the street when I see an Email come in that reads “GA Tracking Issue – Please Help!”, flagged as High Importance. Guess lunch is going to have to wait a while…

12:19 PM – Problem solved! Turns out there were two sets of Google Analytics tracking code on the same page, one urchin.js version and one ga.js version, which is bad news. I then proceed to solve another problem – my hunger.

1:10 PM – I return and find some great discussions starting up on the Yahoo! Web Analytics Forum. It’s really a great forum to check out whenever you can.

1:15 PM – My in person analytics presentation is in 45 minutes. I am very obsessive when it comes to presentations, as I like everything to be perfect, neat, and organized, so I visit our client’s website one more time, and find that they have repaired a bug in their shopping cart that was the focus of one of my main points in the presentation!! I think of a good way to still use this slide in the presentation.

1:38 PM – One of Khrysti’s (Director of Optimized Services) clients is in a bind. They cannot figure out why they are not seeing “yahoo / cpc” or “msn / cpc” in their Google Analytics profile, like they can for “google / cpc”. I reference my latest blog post about Google Analytics URL Coding, and I strut away confidently as I’ve capitalized on another opportunity to tell someone about my Analytics Blog. 🙂

2:00 PM – It’s showtime! Our clients have arrived, and I hand out my business cards and begin with introductions. It’s always great to be able to meet people in person and talk analytics, Site Search, and Shopping Carts to them. This particular client is using both Google Analytics and WebTrends, and they were really concerned about the differences in data between the two, even though they swear that they have everything installed properly. I explained that different analytics packages will always report different values for the same metric, no matter how perfect your installation and coding is.

3:32 PM – I come back and check my own Google Analytics profile for this blog, and I’m surprised to find so much referring traffic from European blogs! I love that someone in Austria and someone in the Netherlands is reading a blog written by someone half-way around the world. I know this because I frequently check my referring traffic reports, to see who is bringing me additional traffic.

4:00 PM – Another one of Khrysti’s clients cannot for the life of them understand why people type in such simple, generic words such as “shoes”, “belts”, and “hats” into their website’s search feature on their online clothing store. They believe something is wrong, broken, or not working correctly. I am pretty sure that their search function is working properly, but I go to their site and double-check with some test searches anyways. After I verify that it is working properly, I pick up the phone and begin to explain to the client that people have much different behavior (and level of tolerance!) when they perform keyword searches on Google or Yahoo vs. performing keyword searches on someone’s website. Again, I direct their attention to my blog by referencing my post about a website’s internal search function.

4:45 PM – My day is starting to come to a close. I like to take a few minutes each day and “spot check” different analytics accounts, just to ensure that everything is still running smoothly and data is being collected and displayed properly. I’m glad I did this, because an important Goal in one of April’s (Director of Strategic Accounts) clients’ accounts has stopped collecting data. After a test on the client’s website, it turns out that the Goal URL has been changed from “thankyou.html” to “thanks.html”. Websites are updated all the time, which is a good reason to routinely double-check your Goals to make sure they are working properly.

5:03 PM – I’m just about wrapping it up here and saying good night to everyone in the office. Out of nowhere, Danielle (my boss) catches me right before I walk out the door. She explains that a new client needs to speak with someone urgently (first-thing tomorrow morning) about what analytics platform they should choose between Omniture SiteCatalyst Hitbox (HBX) or ClickTracks. They also need help in defining new Key Performance Indicators for their executive team, and possibly setting up some custom reporting. I love to think about things like this, especially on off-hours, so I’m glad I have this opportunity.

7:00 PM – Analytics is going to have to wait a while – an episode of Law and Order is on right now that I’ve never seen before. 🙂

All in a day’s work. 🙂

May 1 2008

If you are not testing, then you are wrong!

by MoreVisibility

Back in High School, I was a Lieutenant in the Junior Reserve Officers Training Corps (Go Eagle Battalion!). Class leaders had a lot of fun during uniform inspection time, as well as during drill and ceremony time. We would routinely shout out several commands and instructions to our particular platoons during each class hour, which also included several popular sayings within our corps. These included (but definitely were not limited to):

NINE to the front, and SIX to the rear!” – A reference to the length and distance of your arm swing during marching;
“Get in STEP [Cadet’s Rank and Last Name]!!” – During marching, this would be sounded off to ensure that each cadet’s step would precisely match every other cadet’s step. If every cadet stepped with their right foot, the cadet that stepped with their left foot would stick out like a sore thumb;
“Move the RIFLE around your head, not your HEAD around the rifle!” – My personal favorite during drill and ceremony with our Springfield M-1903’s;
“If you are not double-timing it, you are WRONG!” – You can replace “double-timing it” with any number of different instructions or tidbits of information to convey the message that this was a team effort, and you were wrong if you weren’t participating like everyone else.

We can take that last saying and apply it to testing and experimentation on the web. If you have a website and a marketing plan of any kind, it is imperative that you implement a testing and experimentation plan. Why? Because if you are not testing, you are wrong. In today’s internet world, you absolutely need to have some kind of testing strategy where the ultimate goal is to improve your website’s functionality, your lead acquisition process, and your shopping cart, so that you can have even happier customers, create some more returning shoppers, and ultimately make more money.

For starters, it doesn’t matter what you test – just get your feet wet!

If you’ve been thinking about testing, or if this blog post is the first you’ve ever heard of it, know that for right now, it doesn’t really matter what you test. The mere fact that you going to start testing something – anything – is good enough for now. Get your feet wet and get comfortable and familiar with the idea first, before worrying about what types of testing strategies exist or what standard deviation stands for. Pick anything on your homepage to test for a week or two – that picture of a palm tree, that blue “click here” button, or that first paragraph of text. Pick one of those items (only one for now), and make a change to it, upload it live, and see what effect that has on your traffic and your conversion rate over a week or two. Congratulations – you have just tested something!

This testing idea sounds great, but I wish there was a free tool out there that can help me set-up tests or experiments on my website…

Have no fear – Google Website Optimizer is here! Google Website Optimizer (or GWO for short) just recently came out of Beta, and is now available to everyone on the planet for free. GWO affords you the opportunity to create an unlimited amount of experiments, completely controllable and customizable. GWO goes as far as to offer your technical or website programming team a unique set-up page per each experiment, so that they have every piece of code and every instruction necessary to set GWO up for any page on your website.

What types of Experiments can I conduct with Google Website Optimizer?

There are two different types of experiments:

A/B Experiments – Sometimes also referred to as “A/B Split Testing”, this tests one page on your website up against a different version of that same page, to see which page gives you the best possible chance for an increased conversion rate. Rotating your Ads on Google AdWords evenly is a form of A/B testing in the marketing world. This is the same concept, but for a page on your website.

Multivariate Experiments – Sometimes also referred to as “MVT Testing”, this tests different areas of a page on your website (for example, different headers, footers, or product images), to see which combination gives you the best possible chance for an increased conversion rate. This is actually quite an advanced type of test, but Google Website Optimizer makes it easy for all of us.

How long should I run a test for, and what results will Google Website Optimizer show me?

I like the 15-day rule. With 15 days, you get two full weeks, plus that additional day’s worth of information. This could be longer or shorter, depending on the volume of traffic to your website. However, something in the neighborhood of two weeks should be enough time for a proper experiment.

Google Website Optimizer gives you a “Page Sections” report and a “Combinations” report (specifically for your Multivariate Tests) for you to look at. You’ll be able to view the estimated conversion rate range, in both a numerical form and a sliding bar graph, as well as other fancy statistically-oriented metrics, such as “Observed Improvement”, and “Chance to beat Original / Chance to beat All”, allowing you to very quickly see which page version or which page combination is doing the best job of bringing you more conversions.

What if I run a test between my homepage and a new version of my homepage, but the original homepage beats the new homepage – is it back to the drawing board?

Yes, and no. First of all, you’re going to have to become comfortable with the idea that an original page / original combination beating a newer page or newer combination doesn’t equate to an unsuccessful experiment. If you’re able to conduct a fair and unbiased experiment, then the experiment itself is successful, regardless of the outcome of the experiment. Google Website Optimizer runs fair and unbiased experiments, so rest assured that your experiment will be a successful one.

Now, just because your original homepage beat your new homepage, doesn’t mean that you can’t learn something that you can use in your next experiment. Keep track of what changes were made on the new homepage, and what was different on the new homepage versus the original homepage. If you only make one or two changes, you’ll have a much easier time in keeping track of exactly what’s making the visitors tick and what’s making them leave your site than you would if you completely re-invented the wheel and made several dozen changes.

Other than the homepage, what other types of pages can I experiment with?

The question should really be “what can’t I experiment with?”. You can and you should experiment with all different types of pages – homepages, about us pages, thank you pages, shopping cart pages, order confirmation pages, and so on. GWO lets you run an unlimited amount of different experiments, and you can also run multiple experiments simultaneously with different parts of your website.

Stay Committed!

After you’ve started testing, don’t let the novelty of it wear off. Find a way to make testing and experimentation a part of your job. I know, I know – you’re very busy and you have a lot of work to do, and you can’t possibly imagine putting on yet another hat on. But you JUST have to! Otherwise, your competitors will begin to fly right past you and take your customers away from you. You wouldn’t want that, would you?

Try this: every month, pick 1 thing to test. The “Add to Cart” button, the homepage text, the links on your “Thank You” page…anything. In a few months, you will thank yourself, as you will (hopefully) work towards making your website more attractive to your visitors, which should in turn increase every marketers metric, the conversion rate. Even if your conversion rate doesn’t increase, you will at least have started to learn about your visitors – what they like, what they don’t like, and what they react positively or negatively to – which can only help your business.

March 13 2008

My Google Analytics Wish List

by MoreVisibility

I know, I know…it was just the other day that Google Analytics officially launched Benchmarking and Audio Ads integration. However, I’m greedy, and I already want more!

So, I’ve created a list of things that I want Google Analytics to build into the program, and I’d like to share those with you right now. Some of them may sound like pipe-dreams, while others may actually seem reasonable. For the purposes of this post, I ask you not to burst my bubble, and play along. 🙂

#1 – A “Blogs” Report
If you have a Feedburner account (like our blog does), Google Analytics should be able to synch your GA and your FB (Feedburner) accounts up, like it synchs up AdWords and now Audio Ads. Then, we could see Feedburner-like stats in Google Analytics, such as susbcribers, reach, uncommon uses, and so on. Admit it, it would be cool, wouldn’t it? This report could be found in the Content Section, right underneath “Top Content”.

#2 – Custom Reporting Options
I would love to be able to pick a custom date-range when I create an automatic report. Right now, the only four options to choose from are “Daily”, “Weekly”, “Monthly”, and “Quarterly”. What if I want a bi-weekly / mid-month automatic report? Or what if I want an “every 10 days” report that gets emailed to me automatically? Wouldn’t that be neat?

#3 – A “Reset Your Dashboard” Button
This would be perfect for that one time where you totally screw up your dashboard and you want to start over again with the default dashboard reports.

#4 – An even more customizable dashboard!
This is an extension from request #3. What if I don’t want to have the “Site Usage” window on my dashboard? What if I want each widget on the dashboard to show the top 10 or 15 items, instead of the top 5? How in the world do I get one of the four Google Analytics Views to show up on the dashboard page exactly as I want them (showing the pie chart, the comparison to site average, etc…)?

#5 – Cost-Data Import from other Pay-Per-Click Marketing Programs
Yeah, you probably knew this one was coming sooner or later. This is probably the most requested feature, ever (at least by me, I’ve been requesting it since 2006). This can actually be a manual import, it doesn’t have to be automatic. I’d spend that extra minute on the first of each month and upload my Yahoo! or my Microsoft AdCenter cost data into GA, via a CSV file, so that I could compare that data up against my AdWords data, wouldn’t you?

#6 – Integration with Google Website Optimizer
A logical request, as I already have requested an integration with Feedburner. This makes a lot of sense to me – you would be able to see the results of your A/B or your Multivariate Tests in Google Analytics, and be able to compare your successful combinations or pages against your other current website pages.

#7 – Embed the Map Overlay on your website
This one isn’t necessarily for me – However, I have heard a ton of requests for this particular item. I guess what you can do right now is simply take a screen-shot of the Map Overlay, and save the image and upload it to your site…but maybe there could be some way to provide a snippet of code to be placed on a website?

#8 – Improved Site Overlay functionality!
I completely LOVE the Site Overlay report. However, whenever there is flash present on a site, or there are heavy amounts of dynamic scripts, Site Overlay “breaks” and cannot show any clickstream data. I hope that someday, Site Overlay will be upgraded – and possibly with a few bells and whistles added on to it?

#9 – Bring Back the Lookup Table!
Google Analytics has an advanced filter option called “Lookup Table”, that is currently unavailable. In fact, it has been unavailable for a very long time. Basically, I’d like to play with it, so please bring it back and re-activate it! I actually had a possible use for the Lookup Table a while back.

#10 – Goal re-ordering option
I would love to have the ability to re-order Goals in the same fashion that I can re-order Filters. This way, if I happen to create a Goal in the G1 slot, and then I create a Goal in the G3 slot, I could move the one in G3 to the G2 position, so that I can be as organized as possible, and keep similar goals close together. I hate having to rename and re-configure a goal to have to do this – a lot of times I create a goal or two goals, and later on down the road, another goal becomes available or thought of. Meanwhile, the original goals have already gathered data, so it would be a shame to change the Goal URL and / or Goal Funnel.

© 2021 MoreVisibility. All rights reserved.