How Google searches forms and crawlable websites: blessing or curse?

Marjory Meechan - April 23, 2008

Building a crawlable website is one of the most important things you can do to have success in search engine results pages. After all, if the search engines can’t find your pages, they won’t be listing them. Make no mistake. Google wants to find your pages. Finding every page of the internet is the ultimate goal of all serious web crawler designers and there are numerous articles (both web and print) devoted to their quest to crawl the deep web. Last week Google announced a major step forward in their quest to do just that by searching within the form element on web pages — in other words, searching with the “Search” box on your site. At present, they are restricting this to a few high profile sites, so the average site owner does not have to worry too much but this new development has caused some concern in the internet community.

The main concern of most webmasters on hearing this news was the duplicate content issue. Any SEO friendly website already tries to make their site accessible to Google by providing accessible links. If search engines are going to make random searches using the search button, there could be a lot of duplicate pages created by search queries that don’t have a result. Some search forms like this one shown below give the user pre-set choices to make search easier. If Google tries every possible search that could be created using a search form like this, a great many pages could be created especially if they input potential keywords.

Google claims that sites won’t be penalized for the content found by search engines but some webmasters might want to reconsider the way their sites display content. For example, Google’s Matt Cutts has recommended in the past that pages that consist only of search results not be used as targets for search. The fact that Google can now find these pages for themselves means that while the pages that they find when Google searches forms may not be competing with your real pages, what Google considers a real page might be up for debate. This is just one more reason to create pages with real content and when displaying product lists, always make sure to supplement it with at least one paragraph of plain text content, summarizing the overall theme of the page. Not only will your real pages be distinguished from your plain search pages but it will be much easier to optimize your pages for your targeted keywords.