The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip.
Many webmasters utilize this file to help the search engines index the content of their websites.

If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages. The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results.

That being said, I have seen many occasions where the robots.txt has not been used in the best way possible. For instance, webmasters are prone to make mistakes when installing the robots.txt and the repercussions can be severe. There is a simple instruction that restricts all search engine spiders from crawling the entire site:

User-agent: *
Disallow: /

Without the “forward slash” in the instructions, search engines are granted access to the entire site. So, the inclusion of this one character in the robots.txt can prevent a website from showing in the search engines. There could be many reasons why webmasters would do this intentionally (website is still relatively new and they may still want to tweak certain pages for keyword density etc.), but more often than not, it is a mistake and is usually only realized when the site hasn’t shown up in the search engine indexes for months.

Errors aside, another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap with this simple instruction:

sitemap: http://www.client.com/sitemap.xml (this assumes the xml sitemap is located at the root of the domain).

This also increases spiderability for the search engines. Of course, even though this is a small aspect of the search engine optimization process, if utilized correctly, a robots.txt can be a significant benefit.

Recently, Microsoft released a beta version of an SEO Toolkit that can be integrated with its web server software, IIS 7 (Internet Information Services), which is part of the Windows 2008 Server installation. The toolkit provides several SEO related tools for web developers and web server administrators. It shows that Microsoft is committed to making a website’s content more SEO friendly by offering free tools, an increasing trend for Microsoft, that provide detailed, real-time, analysis of a site’s structure. The SEO toolkit offers a variety of features, including site analysis, robots exclusion, sitemaps and sitemap indexes, all accessible through an easy to use interface integrated within the IIS 7 administration tool.

Web developers and web administrators have been waiting a long time for a tool like this. The SEO toolkit will help identify multiple SEO errors, such as duplicate content, broken links, invalid markup and poorly written title tags, meta descriptions and meta keywords. Traditionally, developers using IIS had to rely on external tools to get this valuable data. Now these tools are available in the same interface where other parts of a website are administered. The SEO toolkit will take a while to catch on, since it is still a beta version for a brand new operating system, I think it’s a great step in the right direction for Microsoft. It definitely

To read more about the Search Engine Optimization Toolkit for IIS 7 please visit http://www.iis.net/extensions/SEOToolkit

Today, I’d like to help reduce your fears and address concerns about implementing a content management system on your website.

You may be thinking about the vast number of pages on your current site, the difficulty of maintaining them and considering how using a Content Managements System could help relieve your headaches. One of your first tasks in going down this road will be matching your needs to a CMS. Here are a few survey questions to help you articulate your needs:

1. How many main sections (applications) does your site have or will you be planning to implement on your new web site (for example, an articles section, a blog, a forum, a store, wiki, CRM, etc.)?
2. Do these sections need to be integrated in any way?
3. How many different content pages do you have (not blog entries, forum or store/product pages)?
4. How often do you make changes to your content in each section of your site?
5. How many people will be administering various parts of your content?
6. What are the skill levels of the people administering the site?
7. How many visitors do you expect in any given hour?

Fear #1: High Cost
While there’s no question that some CMS implementations have been an expensive endeavor for some, it doesn’t have to be that way. If you take an accurate and thorough assessment of the scope of content and features you want to migrate before creating expectations, you will thank yourself for not having to change requirements down the road. Having a well thought out organization of your content and interface (“information architecture” and “user experience,” respectively) before starting to implement your CMS will save you a lot of time and money. Even if you are considering a proprietary CMS and you need some features that aren’t part of the core CMS, find one that is designed to be extended easily. Whether you or your vendor implements the new features, having the features as add-on modules (instead of bolted on to the CMS core) will drastically reduce the development time later on.

Fear #2: Steep Learning Curve
I’m sure you will want to be able to edit your content right away and not have to read the manual in order to make adjustments to your site. The learning curve of your CMS will largely be dependent on the amount of flexibility you need in the system: the more flexibility, generally the steeper learning curve. It is also dependent on the simplicity of the administrative concepts. Consistency in the administration interface also helps to speed your learning curve. Remember that a modular design goes a long way in maintaining simplicity and consistency in both the CMS as a while as well as the administration interface. The good news is that most modern content management systems use this design.

To help reduce the learning curve, it’s a good idea for most people to choose a system that doesn’t have an exorbitant amount of core features over and above their needs. There are hundreds of content management systems from which to find a great match for your organization’s needs.

Fear #3: Not Flexibile
Everybody’s needs are different. Even if you do not pick a CMS that is custom-made from scratch, you may still need a fair amount of flexibility to adjust it to your needs. One of the best ways to attack this fear is to consider systems that have a large user-base and are modular by design. Again, many modern content management systems are designed like this. Usually a large user-base will naturally increase the level of flexibility the CMS has, because all those users have a large number of various needs, and to support all these needs, a successful CMS must be flexible.

Fear #4: It Will Take a Lot of Work to Migrate My Data
If you have a lot of pages and have not profiled your data to see the various formats and systems your data is in, I would suggest you do that. Otherwise, your fear may be unfounded. For example, if you’re using some kind of template system, like a Dreamweaver template, the HTML code surrounding the main content will usually be the same for all your pages. You can then use a search-and-replace process (using a good text editor or a program like RegexBuddy and possibly practicing creating simple regular expressions) to extract the content from your pages, making it faster to put the page’s HTML code into the CMS’s database.

If your content is already in a database, there may be very little work that needs to be done to migrate your data.

Fear #5: It Will Be Difficult to Integrate With Other Systems
Every organization’s needs and situations are different. Not everybody uses the same shopping cart package or the same blog software. The amount of software combinations we find our clients using is huge. However, chances are that other people using the same software you are using have some experience integrating it with other systems. Google searches such as [integrate SOME CMS HERE with SOME APPLICATION HERE] can get you started.

Once you’ve found a good, modular CMS, you should consider writing (or have some one else write) a single module that ties all your other systems to the CMS. This is as opposed to adding code to the core of the CMS for each system you are integrating. All modules written for a specific CMS generally have the ability to communicate with the CMS already built in because they are built from the same framework or foundation. This approach to integration reduces variables, streamlines your development and leaves the interchange between the module and the 3rd party systems as the only thing left to create. All of these reduce development time. Once you have written the interchange between the first third-party system, you will have learned how to interface with the module, and all other interchanges will take less time to code.

Fear #6: Support Will Be Hard to Find
Hopefully you won’t need support for your CMS, but if you have a lot of requirements and many users, eventually a support issue may arise that will be best answered by a live person after you read the manual.

MoreVisibility’s CMS implementations are part of our Optimized Design package, which can be utilized with any amount of optional maintenance hours. If you are not using our CMS, your vendor may provide support. Or, if you’re using a well known CMS, there should be plenty of support resources on the web including forums where you may be able to ask questions. In addition, there may be tutorial DVDs or books available, as well as third-party vendors that specifically support your CMS.

I hope I have addressed your concerns about implementing a content management system. If you haven’t yet, please read my blog series called Content Management Systems and SEO for general information about how they work and what to look out for.