The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip.
Many webmasters utilize this file to help the search engines index the content of their websites.

If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages. The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results.

That being said, I have seen many occasions where the robots.txt has not been used in the best way possible. For instance, webmasters are prone to make mistakes when installing the robots.txt and the repercussions can be severe. There is a simple instruction that restricts all search engine spiders from crawling the entire site:

User-agent: *
Disallow: /

Without the “forward slash” in the instructions, search engines are granted access to the entire site. So, the inclusion of this one character in the robots.txt can prevent a website from showing in the search engines. There could be many reasons why webmasters would do this intentionally (website is still relatively new and they may still want to tweak certain pages for keyword density etc.), but more often than not, it is a mistake and is usually only realized when the site hasn’t shown up in the search engine indexes for months.

Errors aside, another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap with this simple instruction:

sitemap: http://www.client.com/sitemap.xml (this assumes the xml sitemap is located at the root of the domain).

This also increases spiderability for the search engines. Of course, even though this is a small aspect of the search engine optimization process, if utilized correctly, a robots.txt can be a significant benefit.

There’s a new tool for webmasters that was recently unveiled by Google and it’s called “Change of Address”. The tool, which can be found in Google’s newly revamped Webmaster Tools, allows webmasters to tell Google when they are moving to a new domain from their old one. Essentially, it lets Google index the new pages faster to ease the transition of the domain change.

Of course, webmasters still have to follow the usual protocols for redirecting their old pages to the new ones. For instance, making sure all of the 301 permanent redirects are configured correctly to tell both the search engines and users the location of the new pages is still of the utmost importance. Also informing webmasters to update any inbound links to your site still has to be part of the checklist of “moving items”.

What impact will this have on SEO? For one thing, webmasters can now rest assured that their hard work in setting up the new site will not be in vain. Providing they follow the other recommendations for site redirection, Google has provided this as a sort of insurance policy to make sure the user receives the most current search results.

In the future, Google and other search engines may make it even simpler to specify newer domains for webmasters. Search engines in general are getting much savvier in specifying the correct URLs to show in the results pages. This makes it better for both the user and webmasters. Webmasters can now focus on getting to the important business of optimizing the website and not be as concerned as to whether Google is indexing the correct URL.

With Microsoft’s latest search engine, Bing, launching yesterday, there has been a lot of speculation as to whether their “decision engine” will survive. Even with the heavy marketing onslaught reporting to cost over $100 million, the unveiling of this new and improved search experience may have some tough hurdles to overcome.

For instance, Google still controls a major slice of the search engine pie and while it would not be impossible to overthrow the search giant, it will definitely be an uphill struggle. With Microsoft’s “Live” search engine being replaced with “Bing”, they’ve essentially done some clever re-branding and may be able to pull from third place and possibly overtake Yahoo! in the search engine war.

Launched in 2006, Live Search never became the success that Microsoft envisioned. Microsoft blamed it on branding and its distinct lack of marketing. With Bing, Microsoft has added such features as search suggestions as you type, search history and associated search suggestions appearing on the left side of the results page.

As of June 2nd 2009, Bing has already run into some trouble as they have been blocked in China for undisclosed reasons. While this may have little effect on Bing’s overall success, it could be an ominous sign that it may not be an easy road ahead for the young search enterprise.