Articles in The 'Bots' Tag

December 26 2012

Bot traffic in Google Analytics

As I mentioned in my last post, AdWords Bots in Google Analytics (GA), we’ve seen more and more bot traffic registering as visitors in GA. (Bots are automated programs that normally don’t fire JavaScript or images, so this traffic shows up in web server logs, but not in Google Analytics.)

In this post we’re going to discuss bots from Yahoo! and Microsoft, why it’s important and how to identify the traffic and see if it’s affecting your site.

Why is this important? Well as you’ll see, all of this bot traffic comes into your site as Direct traffic, has exactly one pageview and then does nothing, and that is the problem. We have to remember that a visit as just described equals a bounce — which is a bad thing. So as you look at your reports over time, you may wonder why your goal conversion rates or Ecommerce conversion rates from Direct traffic have plummeted while your bounce rates have increased. Part of the answer could very well be bots. And if you don’t account for this traffic in your quest for the analytics intelligence that will turn your site from a business cost to a profit center, you may never get there!

So how do you know if this traffic is affecting your site? Well by looking from 30,000 feet, you may never know — you have to dig deep. So if you haven’t already been digging for answers about your Direct traffic performance, let me walk you through how to identify these bots.

First of all we know that the focus area is Direct traffic that bounces; so the first step is to create an advanced segment to “filter” all of our reports for these visits in Google Analytics Reports.

If you’d like to “play along” as you read this post here is a link to the segment:

https://www.google.com/analytics/web/permalink?uid=BH3Nk-LCSpmv7UQyPXgGDA

This is a view of the service providers in the Network sub-section of the Technology Report that have been the source of our Direct, bouncing traffic.

Why look here? Well there isn’t much to glean from other reports. Content reports are varied. Traffic Sources and Conversions we know, so the best place to try to find some answers is in the Audience section of Google Analytics. Any report here is a good starting place and in this case we can see from the screen shot above that we’ve gotten a lot of traffic directly from Microsoft and yahoo! inc.

So let’s take a closer look at this bouncing, Microsoft and Yahoo! traffic by applying the segments below.

Microsoft:
https://www.google.com/analytics/web/permalink?uid=c8BzLRzjTnmR9JFa48cbsw

Yahoo!
https://www.google.com/analytics/web/permalink?uid=xXjmJTD8RuakRpcLeR9cQA

So the data that points to bots here is pretty straight forward:

100% Bounce Rate

Nearly 100% New Visits

No Java Support

So let’s look closer at the Internet Explorer Traffic:

We can see that most visits are from IE7 and again with no Java Support. (Java Support by itself isn’t necessarily a “bot indicator”, there are other supporting traits that we don’t have the space to include or address in this post.)

While Yahoo! ignores IE, both companies are leveraging Mozilla Agents.

While Microsoft eschews Firefox 3.5:

So what does this all mean? First, it’s highly likely that Google, Microsoft and Yahoo! are using automation to explore websites and on the highest level, that it’s no longer safe to say that bots are not tracked in Google Analytics. As Analysts, Marketers or business owners, we all need to make sure we’re accounting for their presence as we explore analytics data.

In my next post I’ll share some strategies to filter out this traffic and more segments to help you remove the unwanted effects of this traffic.

Tags: advanced segments, Bot Traffic in Google Analytics, Bots, Google Analytics

Posted in: Analytics

December 10 2012

Bot traffic from AdWords in Google Analytics Data

by Theo Bennett

In order for Google Analytics to track visitors; both javascript and images have to be enabled in the visitor’s browser. Most bots, or programs that are written to digest the coding of web pages and collect information, don’t fit that tracking criteria ; therefore, this traffic show ups in web server logs but not in Google Analytics.
Recently there has been an increase in the number of bots that are visiting via real browsers and are able to execute GA code and thereby pollute your Google Analytics data reports. For me, these bots fall into four categories:

Website Monitoring Services — These services continuously check your site to monitor uptime and other things like page load time. (Addressed in this post by a fellow GACP, Blast Media.)
Legitimate Bots other than Google: — This is unexplained bot traffic, but we’ve seen a lot of it recently from Yahoo! Microsoft and Inktomi.
Rogue Bots — Lets face it, any 12 year old can probably write a bot to send “visitors” to your site and wreak havoc on your GA data. (Numbers 2 & 3 here are to be addressed in a future two part post.)
Google AdWords — This is the biggest surprise and as I’ll demonstrate in this post, Google is clearly sending either multiple visitors to AdWords customer pages or they are leveraging a bot.

My interest in this topic began after a colleague inquired about seeing triple the volume of test data expected after creating staged campaigns in AdWords. This data was visible in the Advertising section of Analytics:

Another colleague experienced something very similar when creating new staged campaigns with a small about of URL testing:

After isolating the traffic via an advanced segment using the campaign name of these yet to be launched campaigns, we were able to view the unique characteristics of these visitors.

Most were from the same geographic area, the united states, but suspiciously, city locations were equal to (not set):

Most used the same browser, resolution, and flash version:

As you can see above, 100% of this non-testing traffic bounced.

And interestingly, all of these visits shared the same service provider: google inc..

While we’ve always seen some visitors from google inc.; the spikes in the last few months are different and concerning as to the number of visits and how they can affect data analysis if not accounted for. We’re still evaluating why only some clients see this spike of activity and there are some commonalities we noticed; however, it’s too early to say exactly why this is happening.

As we learn more, we intend to update this post and would like to know about your own experience. Have you seen AdWords bots in your own reports?

If you want to explore your own data, feel free to use the advanced segment below to identify your own traffic from Google. Disclaimer: This segment could also identify the traffic from real Googlers (people that work for Google) that share the same identified service provider.