In this post we’re going to discuss bots from Yahoo! and Microsoft, why it’s important and how to identify the traffic and see if it’s affecting your site.
Why is this important? Well as you’ll see, all of this bot traffic comes into your site as Direct traffic, has exactly one pageview and then does nothing, and that is the problem. We have to remember that a visit as just described equals a bounce — which is a bad thing. So as you look at your reports over time, you may wonder why your goal conversion rates or Ecommerce conversion rates from Direct traffic have plummeted while your bounce rates have increased. Part of the answer could very well be bots. And if you don’t account for this traffic in your quest for the analytics intelligence that will turn your site from a business cost to a profit center, you may never get there!
So how do you know if this traffic is affecting your site? Well by looking from 30,000 feet, you may never know — you have to dig deep. So if you haven’t already been digging for answers about your Direct traffic performance, let me walk you through how to identify these bots.
First of all we know that the focus area is Direct traffic that bounces; so the first step is to create an advanced segment to “filter” all of our reports for these visits in Google Analytics Reports.
If you’d like to “play along” as you read this post here is a link to the segment:
This is a view of the service providers in the Network sub-section of the Technology Report that have been the source of our Direct, bouncing traffic.
Why look here? Well there isn’t much to glean from other reports. Content reports are varied. Traffic Sources and Conversions we know, so the best place to try to find some answers is in the Audience section of Google Analytics. Any report here is a good starting place and in this case we can see from the screen shot above that we’ve gotten a lot of traffic directly from Microsoft and yahoo! inc.
So let’s take a closer look at this bouncing, Microsoft and Yahoo! traffic by applying the segments below.
So the data that points to bots here is pretty straight forward:
So let’s look closer at the Internet Explorer Traffic:
We can see that most visits are from IE7 and again with no Java Support. (Java Support by itself isn’t necessarily a “bot indicator”, there are other supporting traits that we don’t have the space to include or address in this post.)
While Yahoo! ignores IE, both companies are leveraging Mozilla Agents.
While Microsoft eschews Firefox 3.5:
So what does this all mean? First, it’s highly likely that Google, Microsoft and Yahoo! are using automation to explore websites and on the highest level, that it’s no longer safe to say that bots are not tracked in Google Analytics. As Analysts, Marketers or business owners, we all need to make sure we’re accounting for their presence as we explore analytics data.
In my next post I’ll share some strategies to filter out this traffic and more segments to help you remove the unwanted effects of this traffic.
Recently there has been an increase in the number of bots that are visiting via real browsers and are able to execute GA code and thereby pollute your Google Analytics data reports. For me, these bots fall into four categories:
My interest in this topic began after a colleague inquired about seeing triple the volume of test data expected after creating staged campaigns in AdWords. This data was visible in the Advertising section of Analytics:
Another colleague experienced something very similar when creating new staged campaigns with a small about of URL testing:
After isolating the traffic via an advanced segment using the campaign name of these yet to be launched campaigns, we were able to view the unique characteristics of these visitors.
Most were from the same geographic area, the united states, but suspiciously, city locations were equal to (not set):
Most used the same browser, resolution, and flash version:
As you can see above, 100% of this non-testing traffic bounced.
And interestingly, all of these visits shared the same service provider: google inc..
While we’ve always seen some visitors from google inc.; the spikes in the last few months are different and concerning as to the number of visits and how they can affect data analysis if not accounted for. We’re still evaluating why only some clients see this spike of activity and there are some commonalities we noticed; however, it’s too early to say exactly why this is happening.
As we learn more, we intend to update this post and would like to know about your own experience. Have you seen AdWords bots in your own reports?
If you want to explore your own data, feel free to use the advanced segment below to identify your own traffic from Google. Disclaimer: This segment could also identify the traffic from real Googlers (people that work for Google) that share the same identified service provider.
Google Inc. segment:
And here is one to exclude traffic from google inc.
from your reports: