Building a crawlable website is one of the most important things you can do to have success in search engine results pages. After all, if the search engines can’t find your pages, they won’t be listing them. Make no mistake. Google wants to find your pages. Finding every page of the internet is the ultimate goal of all serious web crawler designers and there are numerous articles (both web and print) devoted to their quest to crawl the deep web. Last week Google announced a major step forward in their quest to do just that by searching within the form element on web pages – in other words, searching with the “Search” box on your site. At present, they are restricting this to a few high profile sites, so the average site owner does not have to worry too much but this new development has caused some concern in the internet community.

The main concern of most webmasters on hearing this news was the duplicate content issue. Any SEO friendly website already tries to make their site accessible to Google by providing accessible links. If search engines are going to make random searches using the search button, there could be a lot of duplicate pages created by search queries that don’t have a result. Some search forms like this one shown below give the user pre-set choices to make search easier. If Google tries every possible search that could be created using a search form like this, a great many pages could be created especially if they input potential keywords.

Google claims that sites won’t be penalized for the content found by search engines but some webmasters might want to reconsider the way their sites display content. For example, Google’s Matt Cutts has recommended in the past that pages that consist only of search results not be used as targets for search. The fact that Google can now find these pages for themselves means that while the pages that they find when Google searches forms may not be competing with your real pages, what Google considers a real page might be up for debate. This is just one more reason to create pages with real content and when displaying product lists, always make sure to supplement it with at least one paragraph of plain text content, summarizing the overall theme of the page. Not only will your real pages be distinguished from your plain search pages but it will be much easier to optimize your pages for your targeted keywords.

Image search relevance is not something that we think about very often, but as Karen Umpierre noted recently, image search is getting smarter. Google can look inside images and tell the difference between faces and non-faces … well, usually. Occasionally a non-face slips in but it’s pretty impressive. This brings up the question of the future of search. Will there come a day when we won’t have to put 200 words of text on a page for it to rank well for relevance in a search query? The answer may be yes because new search engine technologies are emerging that take search into the images themselves. Instead of image search using text, we have visual search using images.

Maybe the most exciting new visual search idea is Nokia’s wireless Point & Find where a user just takes a picture and gets relevant information about the image based on information on the internet – including location, price, etc. While this is very much an emerging technology, it could very well change the way people shop. Imagine seeing some shoes you like at a party and being able to find them using your wireless device’s built-in search. Nokia claims this technology is at least three years away, but in the meantime, there are some visual search engines available now.

One that deserves special mention is Like.com because it is the first engine that is actually using visual search to help shoppers find what they want. For example, let’s say I want to buy some shoes. I go to the shoes section of Like.com and find a style that I like. Then I just click on the button marked Visual Search. This gives me a display of shoes that all resemble the shoes that I chose grouped by “overall matches”, “style matches”, “brand matches” and “color matches”. The most interesting aspect is the opportunity to refine my search by selecting a portion of the original image.

I just draw a box around the part of the image that I most want to match and Like.com gives me a new result. Like.com has just recently launched a new site after a lengthy beta period so we’re interested in seeing how well the new site is received. Will shoppers take to it? One problem may be that, as we noted with Google’s face search, the results aren’t perfect. For example, one of the top ten results for my search for matches to that spiky heel was this wedge heeled number. Of course, it is a high wedge – sort of spiky as wedges go and roughly the same shape as the heel of the shoe I selected so maybe this is just the visual search algorithm’s interpretation of my query. This brings me to the point of my post today – image relevance.

Just because a search engine uses a different point of reference (shape, color, style as opposed to words) doesn’t mean that we can stop worrying about relevance. If visual search becomes more prevalent, search engine optimization may actually become even more complicated with considerations of image quality, angle and other visually based factors coming to the fore. Furthermore, even Like.com relies on linguistic navigation to subsidize their product search and this is unlikely to change. So, if you were waiting around for the image search technology to get better before you optimized your site for keyword relevance, we would recommend that you not wait because whether it’s images or text, making your pages relevant to search queries will always matter.

Site sculpting with the nofollow attribute is slowly becoming a hot topic in the SEO world and the information that is being put around is not that clear. The nofollow link attribute was designed to relieve beleaguered forum owners and other open source content sites from the deluge of link spam that was clogging up their forum and general information pages. In a previous blog post on the true purpose of the nofollow link attribute, we discussed the correct usage of this attribute. So far, it’s worked pretty well for that purpose but as with all good things, there may be a dark side.

Late last year, Matt Cutts implied that a pro-active SEO use of the nofollow attribute could result in better PageRank for pages on your site leading to a heightened interest in this new SEO technique. A good visual explanation of site sculpting can be found here: http://www.evisibility.com/blog/no-follow-tag/. As recently as this week, participants in the Organic Listings forum at SES New York were recommending this new technique for improving the rankings of important pages of your site.

All this reminds me of a friend of mine who was convinced that he could make his back pain go away by gluing fridge magnets to his back. Magnetic fields may really have a medical use but so far as I know, nobody has been able to show exactly how to paste the fridge magnets on your back to maximum effect so he really had no idea whether he was using them correctly or not. Therefore, he was either doing nothing whatsoever for his back (the most likely possibility) or he could potentially be doing damage.

Site sculpting seems to me to be a little like this and a quick review of the most recent site sculpting buzz shows that I’m not alone in my confusion over the best use of this new SEO technique. I’m not endorsing all these opinions - just showing that there is some difference of opinion. One major problem is that not all search engines interpret the nofollow attribute exactly the same way:

How Google interprets the nofollow attribute:
http://www.google.com/support/webmasters/bin/answer.py?answer=81749&topic=8522

How Yahoo interprets the nofollow attribute:
http://help.yahoo.com/l/us/yahoo/search/indexing/nofollow.html
How MSN interprets the nofollow attribute – this is not explicitly mentioned in MSN HelpCentral but this was their original announcement on the topic:
http://blogs.msdn.com/livesearch/archive/2005/01/18/nofollow-tags.aspx.

These are just the major search engines. As far as we know, other engines like Ask.com do not respect it at all. In fact, there are still many questions about how search engines interpret the nofollow attribute.

All in all, we’re a little suspicious of the claims that Matt Cutts is endorsing the practice. Generally, Matt Cutts doesn’t promote techniques that could potentially manipulate search engine algorithms. This makes us worry that it won’t work and we will have wasted precious SEO time and effort. Or, worse, it will work but not to our advantage.As a result, we’re recommending that if a site owner wants to try it, they should be very careful only to apply it to links to pages that really and truly are unimportant and definitely do not need to be indexed.

The bottom line is that, as always, the best way to optimize your site is to only provide content and links that are valuable to your visitors. The homepage is the most valuable real estate on the site and only the most important links should be found there. If there is a link on your homepage that you are thinking of adding a nofollow attribute to, then maybe a better question would be why is that link there in the first place? In other words, instead of using the nofollow attribute to sculpt your site, try using your main navigation. In the end, it’s more durable and doesn’t depend on the ever changing whims of search engine algorithms.