While writing several blog posts and documentation, I often have used example.com to stand in for any domain name. One of the Internet standards established by the Internet engineers circa 1999 set aside example.com (as well as example.org and example.net) for documentation purposes. So if you were to click on a link to http://www.example.com in my post, you wouldn’t see an actual web page. Click on this link to see for yourself.

I’d like to demonstrate a fun little trick you can use to amaze your friends.

The page you see is when you go to http://www.example.com is completely indexable by the search engines. There’s not a lot of content, but you would think that the engines will have indexed the content exactly as your browser shows it to you. It turns out that there is a robots.txt file that blocks all spiders from all content inside www.example.com. (If you ever forget how to create a basic robots.txt file, you can use this one as a guide.) Alright, now for the punch line. Let’s see what the search engines really have indexed for http://www.example.com. Go to www.google.com and type “site:example.com” (without the quotes). What do you see? If you see only one result, click on the link: repeat the search with the omitted results included.

I see 10,400 results now. There are pages like example.com/blah/ and www.example.com/concepts. The Google search results page does not have links to the cached version for any of these results, unfortunately, so we can’t see what exactly Google has indexed from these pages, but we can go to the page ourselves. Well, I tried that, and every page I go to replies back with “Not Found.” It’s logical to conclude that those pages never existed, but also notice some of the results have been crawled by Google in the past few hours. Impossible, no?

You can try this search on other search engines too.
My feeling on this strange phenomenon is that it could either be Google’s own testing or other people testing or somehow tricking Google into adding these pages to its index. It may be relegated to certain data centers as well.

Whatever is causing this, I’m sure Google knows about it, but doesn’t feel the need to do anything about it. This phenomenon may also get you thinking about how search engines are supposed to work.

Web page optimization can be daunting for the beginner so we’ve put together five easy steps to optimizing a page for search engines.

1. Identify your goal. (This is always the first step in anything, but we’re going to mention it anyway). Remember that ranking number one is great but at the end of the day, the ultimate goal is acquiring customers, so choose the keyword carefully - which leads us to the next step.

2. Research the keyword target and make sure the keyword is right for you.

* Does anybody ever search for that keyword? It doesn’t do any good to rank number one if nobody ever looks for it.

* Is that the word people would use to find my product or service?

* How competitive is the keyword? Who is your competition?

For example, an exact match search for “blue widgets” reveals that about 22,000 other sites are relevant for that phrase while “big blue widgets” displays only about 40. It will be much easier to get the number one spot for big blue widgets. If a keyword is too competitive, consider choosing a longer phrase containing the keyword. Ranking on page 1 for a lower traffic key phrase will bring more traffic than ranking on page 5 for a high traffic keyword.

3. Choose a page to target that key phrase. Which page you choose will depend on a couple of factors:

* Page Content - the more precisely matched the page content is to the keyword target, the more likely that a visitor will click on your link in the search results and buy something once they are there.
* Competitiveness of the keyword - If the keyword is highly competitive, you may need the ranking power of your homepage.

Whatever page you choose, make sure it contains clear information on how to get the product. Ranking well for the keyword doesn’t do any good if the target page doesn’t convert them into a customer.  (Did I say that already? Remember it. It’s important.)

4. Check out your competition for the keyword.

* What’s their current keyword density?
* Does the keyword appear in all of the important places, title tag, description tag, keywords tag, headline, etc. on the competitor’s page?
* How many backlinks do they have to their page and what kind of anchor text appears on those backlinks?

5. Optimize your page for the keyword.

* Put the keyword on the page in all the important places.
* Target a keyword density at or slightly above (or below) that of other top ranking sites.
* Get inbound links to the page – ideally with the keyword in the anchor text.

Alert readers will have noticed that all the steps listed here are really just extensions of the first step – identify your goal. Identifying your goal is definitely the most important step in web page optimization. The second most important step isn’t really a step but it’s still crucial. Monitor your progress and not only for how your page ranks in search engine results pages.

Compare the ranking in search engine results pages before and after and then compare the difference in traffic before and after your listing appeared.

* Make sure you give search engines enough time to crawl and index your pages. It can take up to a couple of weeks after they have crawled your page for changes to produce results.

Compare customer conversions from the page before and after.

* Are your new visitors sticking around to become your customer or are they coming to your pages and just “bouncing away”?

Fierce competition and shifting search engine algorithms make web page optimization an ongoing process. If the results aren’t what you hoped, then maybe this wasn’t the best keyword for you or maybe the page needs a little more tweaking for keyword density or backlinks. You can always benefit from another good backlink. Maybe the page is fine for search engines, but needs to be optimized for visitors. Identifying goals, taking a step-by-step approach, monitoring your progress and never giving up are the keys to success in web page optimization.

Site sculpting with the nofollow attribute is slowly becoming a hot topic in the SEO world and the information that is being put around is not that clear. The nofollow link attribute was designed to relieve beleaguered forum owners and other open source content sites from the deluge of link spam that was clogging up their forum and general information pages. In a previous blog post on the true purpose of the nofollow link attribute, we discussed the correct usage of this attribute. So far, it’s worked pretty well for that purpose but as with all good things, there may be a dark side.

Late last year, Matt Cutts implied that a pro-active SEO use of the nofollow attribute could result in better PageRank for pages on your site leading to a heightened interest in this new SEO technique. A good visual explanation of site sculpting can be found here: http://www.evisibility.com/blog/no-follow-tag/. As recently as this week, participants in the Organic Listings forum at SES New York were recommending this new technique for improving the rankings of important pages of your site.

All this reminds me of a friend of mine who was convinced that he could make his back pain go away by gluing fridge magnets to his back. Magnetic fields may really have a medical use but so far as I know, nobody has been able to show exactly how to paste the fridge magnets on your back to maximum effect so he really had no idea whether he was using them correctly or not. Therefore, he was either doing nothing whatsoever for his back (the most likely possibility) or he could potentially be doing damage.

Site sculpting seems to me to be a little like this and a quick review of the most recent site sculpting buzz shows that I’m not alone in my confusion over the best use of this new SEO technique. I’m not endorsing all these opinions - just showing that there is some difference of opinion. One major problem is that not all search engines interpret the nofollow attribute exactly the same way:

How Google interprets the nofollow attribute:
http://www.google.com/support/webmasters/bin/answer.py?answer=81749&topic=8522

How Yahoo interprets the nofollow attribute:
http://help.yahoo.com/l/us/yahoo/search/indexing/nofollow.html
How MSN interprets the nofollow attribute – this is not explicitly mentioned in MSN HelpCentral but this was their original announcement on the topic:
http://blogs.msdn.com/livesearch/archive/2005/01/18/nofollow-tags.aspx.

These are just the major search engines. As far as we know, other engines like Ask.com do not respect it at all. In fact, there are still many questions about how search engines interpret the nofollow attribute.

All in all, we’re a little suspicious of the claims that Matt Cutts is endorsing the practice. Generally, Matt Cutts doesn’t promote techniques that could potentially manipulate search engine algorithms. This makes us worry that it won’t work and we will have wasted precious SEO time and effort. Or, worse, it will work but not to our advantage.As a result, we’re recommending that if a site owner wants to try it, they should be very careful only to apply it to links to pages that really and truly are unimportant and definitely do not need to be indexed.

The bottom line is that, as always, the best way to optimize your site is to only provide content and links that are valuable to your visitors. The homepage is the most valuable real estate on the site and only the most important links should be found there. If there is a link on your homepage that you are thinking of adding a nofollow attribute to, then maybe a better question would be why is that link there in the first place? In other words, instead of using the nofollow attribute to sculpt your site, try using your main navigation. In the end, it’s more durable and doesn’t depend on the ever changing whims of search engine algorithms.