Clear Google Analytics of Bot Traffic Using User Agent

Theo Bennett - July 13, 2020

As we reported earlier this year, the deprecation of the Service Provider dimension on February 4th, 2020 broke a key method of fighting bot traffic in Google Analytics (GA)

You can follow the link above to learn more about how this valuable dimension was used and why it was important. In this post, I’ll walk you through one of the solutions I mentioned: Capturing the User Agent as a custom dimension in Google Analytics.

Why Capture the User Agent?

Some well-behaved bots have unique, identifiable user agents that allow you to distinguish between real, breathing users of your site and a bot that may be scraping content or checking to see if your site is up. Bot traffic in your Google Analytics data is not going to help you increase revenue but rather will obfuscate real user behavior. If we capture User Agent data, we can analyze it, identify bots (in some cases) and filter these sessions out of data views used for analysis.

Create the Custom Dimension

Go to the property tree in the admin section of your Google Analytics account and select Custom Dimensions.

Add a session scoped custom dimension and name it something that makes sense, like “User Agent” and remember the index number.

Modify Google Tag Manager (GTM)

Now that the custom dimension is ready to receive data, configure GTM to pass this data into GA. First add a JavaScript Variable of navigator.userAgent.

Next, modify your GA Variable and add the custom dimension with the index number (7 in this example) and the name of the variable you created.

Publish the container and the user agent will collect in GA.

Analyze and Filter

Over time, you may begin to see traffic that you can filter out from your analysis views.  For example in the screenshot below, (direct traffic with a secondary dimension of “User Agent” applied, then filtered for 100% bounce rate), we see that clearly the first entry is an automated check from Hubspot that could be removed. Just make sure you collect enough traffic before you filter anything out and always have a view on which no filters are applied.

By using this tactic, you can keep your data free from the clutter of bot traffic and better understand and meet the needs of real site users.