A New Robot Comes to Town: msnbot/1.1

This week, MSN Live Search announced changes to their robot including a new name: msnbot/1.1. Currently, the name change just applies to their main robot, but look forward to more new robots from MSN in the coming months.

The new robot comes equipped with features designed to put less of a load on servers while still collecting the maximum number of new pages. Some of the new features include:

• HTTP Compression: This lets the server download documents compressed with gzip or other formats to the search engine robot instead of the full documents. Because compressed documents are smaller, this reduces the amount of time it takes to access all the pages of a site.
• Conditional Get: This feature allows the robot to check to see if the page has changed since the last time they were there and only download it if it has. Basically, the robot sends a request asking if the page has changed since the last date it was there and if not, it moves on.

These features are great because it means that their robot will waste less time looking at pages it already has and more time looking for new pages. This should result in more pages of a site being indexed – particularly in the case of large sites and less time spent crawling around websites using up bandwidth and putting extra pressure on servers.

This was good news for webmasters who have complained that MSN’s robot was taking up too much bandwidth while crawling their site, but the technology is not new to web hosts. The other search engines have been employing these methods for some time to reduce the load they place on web servers. Yahoo has supported both these technologies since 2005. Ask supports http compression and Google supports conditional get and other methods to reduce bandwidth already. This just shows that MSN is still playing catch-up with the other major search engines. Still, it’s welcome news and MSN has been kind enough to provide us with a tool to check if server settings support http compression and conditional get. For more information on this, just check out the Live Search blog and put out the welcome mat for msnbot/1.1.