As we (or most of us) know, we do not want the search engine spiders crawling around and indexing certain parts of our website. For instance, it is probably not intelligent to have spiders crawling the secure port of the server, i.e. secure parts of our websites (https://).
We also don’t want to have the Googlebot or the Bingbot crawling and indexing pages that would be a poor choice as a target for search, like PDF files, Excel documents of images. This is where the robots.txt file comes in.

The robots.txt file, uploaded to the root directory of the site (www.example.com/robots.txt), tells the spiders which directories and pages to skip. Why do we want this? If someone were to find a PDF file or a Flash file in a search result and were to click on it, those types of documents generally don’t contain links leading back to the rest of the site and can be a “dead-end” for both the the search engines and for the user. A simple “Disallow” instruction in the robots.txt file will prevent non-SEO friendly pages from possibly showing up in search results for your desired keyphrases:

 

I love it when art and science come together. Maybe that is why I am so enthusiastic about SEO and the possibilities of the internet in general. Case in point, the other day I found the periodic table of SEO, courtesy of Search Engine Land. What this little piece of scientific art is showing is a formula for SEO success, based on ranking factors that search engines look for when crawling your website.

The table highlights fundamental ranking factors for on-site and off-site optimization strategies. Likewise, the document sheds light into search engine violations and any blocking by users, via Google’s newly released hiding feature.

The table offers a numerical breakdown of the factors in the upper right-hand corner of each element – a spinoff of the traditional Periodic Table. The numbers (1-3) are meant to indicate the level of importance of each element with “1″ being least important and “3″ representing the highest.

Some of the listed elements are as follows:

On-Page SEO Ranking Factors:

- Content Quality and Research – are your pages well written and has keyword research been done?
- HTML Tiles, Description, and Keywords (Meta Data) – does your meta data contain the keywords and do they describe the page?

Off-Page SEO Ranking Factors:

- Link Quality – are your links from trusted and reputable websites?
- Trust/Authority – do your links and shares make your site trustworthy?

Violations:

- Thin Content – is you content more generic and lacking substance?
- Keyword Stuffing – are you excessively stuffing keywords in your content?

This document is both extremely informative and creative. The challenge is following it and optimizing your website to the letter. That, like the SEO document, is both an art and a science.

Another useful instrument in Google Webmaster Tools is the Sitemap section in Site configuration>>Sitemaps:

 
      Figure 1 - Google Webmaster Tools “Sitemaps” Section

You can check your submitted XML Sitemaps in this section and identify which pages from them Google has managed to index. You can also tell if Google had trouble accessing the Sitemap by seeing if there is a checkmark in the “Status” field. That being said, why do we even want to do this? If a site is already “crawlable” in Google’s eyes, why do we need to perform this extra step?

A Sitemap is a true representation of a website’s structure in that it allows Google to access every page that you wish to have crawled, indexed and potentially be a target for search. It also helps augment Google’s crawling and indexing by allowing Google to crawl pages that they would not be able to access otherwise, such as JavaScript enabled links or links using Flash. Once all of the pages from your site that you want Google to know about are included in the Sitemap (50,000 maximum URLs in one Sitemap and 10MB uncompressed file size limit), upload it to the Sitemap section in Google Webmaster Tools by clicking on the “Submit a Sitemap” button:

 
        Figure 2 – “Submit a Sitemap” Button

Clicking on the above button reveals a field to enter the physical location of the XML Sitemap, which is usually in your website’s root directory, for instance: www.example.com/sitemap.xml.

Keep in mind that the specific numbers reported in the Sitemaps section of Webmaster Tools only apply to the URLs you submitted in your Sitemap(s), not the amount of pages you actually have in the index; there will always be a discrepancy between the “URLs submitted” and “URLs in web index”:

 
       Figure 3 - URLs in Web Index

In fact, it is rare that the number of URLs reported in both sections will be the same.  There could be discrepancies because of restrictions on a lot of files in your robots.txt or just duplicate pages that Google has decided not to index.
Just ensure that URLs in your Sitemap are the “canonical” URLs (www or non-www for example). If there are URLs you care about that aren’t in your Sitemap, just add them in and re-submit. Many times, web developers will add multiple pages to a site and forget to update their Sitemaps. This can be problematic if the new pages are not well interlinked on the site. Remember, your site can have the most optimized pages ever created, but all of your hard work will be in vain if Google doesn’t know about them!