This week, MSN Live Search announced changes to their robot including a new name: msnbot/1.1. Currently, the name change just applies to their main robot, but look forward to more new robots from MSN in the coming months.
The new robot comes equipped with features designed to put less of a load on servers while still collecting the maximum number of new pages. Some of the new features include:
These features are great because it means that their robot will waste less time looking at pages it already has and more time looking for new pages. This should result in more pages of a site being indexed — particularly in the case of large sites and less time spent crawling around websites using up bandwidth and putting extra pressure on servers.
This was good news for webmasters who have complained that MSN’s robot was taking up too much bandwidth while crawling their site, but the technology is not new to web hosts. The other search engines have been employing these methods for some time to reduce the load they place on web servers. Yahoo has supported both these technologies since 2005. Ask supports http compression and Google supports conditional get and other methods to reduce bandwidth already. This just shows that MSN is still playing catch-up with the other major search engines. Still, it’s welcome news and MSN has been kind enough to provide us with a tool to check if server settings support http compression and conditional get. For more information on this, just check out the Live Search blog and put out the welcome mat for msnbot/1.1.