How do the search engines know how to find what it is that you are looking for? When you enter a query, or a series of words or a phrase into Google, MSN or Yahoo, how do they go about giving you sites that relate to your search?

Let’s try to answer this in a straightforward and somewhat simplified way. When it comes to search engine optimization, or SEO, there is usually mention of complex algorithms and predictive analytics. Let’s see if we can boil things down to a basic, real-world example.

Say you have a website, www.example.com. Typically, you would like for Internet users to find your site and to read more about your products or services. You have your content, titles, descriptions, and keywords, you’ve tackled your in-site linking and inbound linking, and you may even have a social media marketing plan in place. But how does it all come together on the back end? If you are doing everything right in terms of SEO best practices, why and how do your search engine rankings change?

At the present moment there are well over a billion pages of index-able content on the Internet. The search engines act as a way in which we can sort through all of that information, and, in turn, use that information to answer a question. The search engines collect and categorize information so they can help to answer the most basic question, that of relevance. This is how your query, the search term we mentioned earlier, relates to the information contained in the search engines’ massive databases. The question is really a matter of determining which web pages are most relevant to the terms A, B, and C.

Having over a billion pages to work through, the search engines have to manage that information in a way which ensures that less relevant information doesn’t appear at the forefront of the search engine result pages (SERPs). This isn’t part of some conspiracy – it’s a matter of trying to make the results as useful as possible, hence the focus on high relevance versus low relevance.

But how do the search engines determine what’s relevant? For this SEO blog post we’ll define “relevant” as being those web pages that have the terms which most closely match the words (keywords) the web searcher typed into the search engine.

This relevance is determined by the search engines roughly assigning a score (as in the case of Google it is PageRank), which is also how websites are listed in the SERPs. Obviously, with the vast number of websites and web pages, there is a wide variety of ways in which to measure and score relevance, some or all of which may be employed by the search engines. As these algorithms are proprietary to Google, MSN, and Yahoo, we don’t know exactly what is being used or in which way, but we do have first-hand experience, sound anecdotal evidence, as well as a wealth of research and observation to come to one conclusion when it comes to search engine relevance.

Content is king and the text is the thing. What can be considered as being the text on your website? It’s what appears in the title tags, URLs, anchor text, image alternate text, the comments (if you have a blog), the description meta tag and keywords meta tag, and in the formatted (or unformatted) visible text areas of a page. You can cultivate the relevance of pages on your site through in-site linking. The text as found on one page can be supplemented by information that is somehow associated with that page (as in a link) as well as related pages which link to that page.

Why is text so important to determining relevance? Much like a human visitor to your website uses the text on the page to figure out what the site is about and where to navigate to next, the search engines do the same. The placement of the visible text, as well as how it is emphasized and used, help the search engines understand what the pages on your site are about. Building keyword density and using keywords with prominence can help to assert the page’s relevance.

Changing the text in the pages changes the relevance. This is why there are changes to rankings and positioning on the SERPs. Even changing the links, images, or even something off site, such as directory descriptions, can tell the search engines that they should re-evaluate your site to ensure that it is still relevant to the previously associated terms or perhaps has greater relevance to a new set or words of phrases.

The thought to keep in mind is that you aren’t the only one changing or optimizing text – your competitors are doing the very same with their web content. Changes made by other websites within your industry or space can indirectly influence your relevance. This is because the search engines collect and compile all that they know about all the websites and web pages they find as they crawl the Internet, and relay this information back to a search engine user based on how all of this information relates to his or her query. So as you optimize www.example.com, www.example.net and www.example.org are likely performing similar activities. Refining and targeting your content by using unique and relevant keywords, as well as keeping a level of freshness for your content and cultivating trusted and relevant inbound links to your site are just some of the ways in which you can help your website to keep its relevance in the search engine algorithms. Content remains king and relevance and usefulness are the underlying forces which ensure content will remain supremely important. Understanding how important it is to the search engines will help both your SEO efforts and your site to grow.

It’s been a while since we’ve talked about URL canonicalization on our blog, so I’ll quickly review what it is and then talk about filename canonicalization and how it can affect your SEO endeavors.

Canonicalization is something done to your site to help ensure that content from one specific URL in your site does not show up under another URL. This is a type of a duplicate content issue. If the engines index more than one copy of some specific content, from separate, full URLs (those containing all necessary parts), they will be forced to divide the “strength” of that content between the URLs. That reduces the “strength” of all URLs involved and reduces the chance that one of your URLs will show up in a search results page for a given search term. Most of the time, the issue will arise when both www.example.com and example.com (and all other pages of the site) show the same exact contents.

Having un-canonical host names is one way that duplicate content can become a problem for your site: a search engine indexes content from your home page at www.example.com and indexes the same exact content at example.com. While on the subject of un-canonical host names, you should also know that it is possible to have un-canonical protocols in your URLs. In a complete URL like http://www.example.com/about-us.html, the “http” is known as the protocol. If search engines happen to index the same content from https://www.example.com/about-us.html, you may have just stepped into a duplicate content issue due to un-canonical protocols.

Now let’s say you have a store (or any other web-based application) on your site that is accessed from http://www.example.com/store. Generally, your ecommerce program would be located in a folder on your server called ’store’ (and usually, URLs of your store’s products are based on that URL: http://www.example.com/store/blue-widgets.html). It’s quite possible that a search engine could index content at http://www.example.com/store/index.php and at http://www.example.com/store/ and count them as duplicate content. This could happen if another site links directly to one of those URLs and your site links to the other URL. This is an example of what I call a file name canonicalization issue.

Those are basically all the canonicalization issues you could encounter. However, often times, you’ll find a site that has a combination of canonicalization issues. For example, all of the following URLs are different, but will all have the same exact content:
http://www.example.com/kitchen-sink
http://www.example.com/kitchen-sink/default.html
http://example.com/kitchen-sink/
https://example.com/kitchen-sink

Perhaps you’re wondering if there would be a duplicate content issue between the following two URLS:
http://www.example.com/blog
http://www.example.com/blog/

The answer will most likely depend on whether ‘blog’ is an actual folder on your server or a file. (Your web server or blog, or any other web-based application may be setup in a way that does not require file extensions such as .html.) Most web servers (the software on the hosting company’s computer that waits for and then responds when someone asks for a web page on your site) work the same way in this situation. If ‘blog’ is an actual folder on the web server, then whenever someone asks for http://www.example.com/blog (without the trailing forward slash), the web server automatically 301 redirects to http://www.example.com/blog/. (This is the correct redirect type for this situation, by the way, and this is the reason that this should not cause a duplicate content issue.) If someone asks for http://www.example.com/blog/ (with the trailing forward slash), the web server doesn’t redirect at all. I’ll talk more about this in my next post.

Marjory Meechan, in her blog How to Resolve the Canonicalization Issue without Access to your Server, discussed un-canonical host name issues and one way that can be used to fix this. Please read my next post in which I discuss the causes of these canonicalization issues and suggest more dependable methods to eliminate them, provided that you have some level of server access.

Ever since Google introduced sitelinks, website owners and webmasters have had lots of questions about how these work. At Google’s recent June Tune Live Chat, the question arose again as to why some sites have these links and how they can control them. Here is an excerpt from the transcript of the Q and A section of the chat with Google’s answer to the question:

Q: Will we ever have control over sitelinks to bring searchers to the better pages on our site, vs the ones Google thinks are the best pages?
A: Sitelinks are automatically generated by our algorithm and are meant to help users navigate your site. While you cannot opt into having sitelinks, you are able to block sitelinks using Google Webmaster Tools.

Later on during the audible portion of the chat, in answer to this question:
Does the design of my site affect how these links are compiled? What can I do to help Google compile better sitelinks?

Bergy Berghausen replied that site architecture can affect your chances of getting sitelinks:

“Having a very simple html based navigation is the best way that we can tell how your site is organized and we try to guess based on where we think people are going on your site and how your site is structured.”

While we can’t tell you how to get the sitelinks, we can tell you how to get rid of them. If Google is linking to a page of your site that you would rather not have in the results pages, you can use the sitelink tool in Google’s Webmaster Tools to tell Google to remove the sitelink.