How do the search engines know how to find what it is that you are looking for? When you enter a query, or a series of words or a phrase into Google, MSN or Yahoo, how do they go about giving you sites that relate to your search?

Let’s try to answer this in a straightforward and somewhat simplified way. When it comes to search engine optimization, or SEO, there is usually mention of complex algorithms and predictive analytics. Let’s see if we can boil things down to a basic, real-world example.

Say you have a website, www.example.com. Typically, you would like for Internet users to find your site and to read more about your products or services. You have your content, titles, descriptions, and keywords, you’ve tackled your in-site linking and inbound linking, and you may even have a social media marketing plan in place. But how does it all come together on the back end? If you are doing everything right in terms of SEO best practices, why and how do your search engine rankings change?

At the present moment there are well over a billion pages of index-able content on the Internet. The search engines act as a way in which we can sort through all of that information, and, in turn, use that information to answer a question. The search engines collect and categorize information so they can help to answer the most basic question, that of relevance. This is how your query, the search term we mentioned earlier, relates to the information contained in the search engines’ massive databases. The question is really a matter of determining which web pages are most relevant to the terms A, B, and C.

Having over a billion pages to work through, the search engines have to manage that information in a way which ensures that less relevant information doesn’t appear at the forefront of the search engine result pages (SERPs). This isn’t part of some conspiracy – it’s a matter of trying to make the results as useful as possible, hence the focus on high relevance versus low relevance.

But how do the search engines determine what’s relevant? For this SEO blog post we’ll define “relevant” as being those web pages that have the terms which most closely match the words (keywords) the web searcher typed into the search engine.

This relevance is determined by the search engines roughly assigning a score (as in the case of Google it is PageRank), which is also how websites are listed in the SERPs. Obviously, with the vast number of websites and web pages, there is a wide variety of ways in which to measure and score relevance, some or all of which may be employed by the search engines. As these algorithms are proprietary to Google, MSN, and Yahoo, we don’t know exactly what is being used or in which way, but we do have first-hand experience, sound anecdotal evidence, as well as a wealth of research and observation to come to one conclusion when it comes to search engine relevance.

Content is king and the text is the thing. What can be considered as being the text on your website? It’s what appears in the title tags, URLs, anchor text, image alternate text, the comments (if you have a blog), the description meta tag and keywords meta tag, and in the formatted (or unformatted) visible text areas of a page. You can cultivate the relevance of pages on your site through in-site linking. The text as found on one page can be supplemented by information that is somehow associated with that page (as in a link) as well as related pages which link to that page.

Why is text so important to determining relevance? Much like a human visitor to your website uses the text on the page to figure out what the site is about and where to navigate to next, the search engines do the same. The placement of the visible text, as well as how it is emphasized and used, help the search engines understand what the pages on your site are about. Building keyword density and using keywords with prominence can help to assert the page’s relevance.

Changing the text in the pages changes the relevance. This is why there are changes to rankings and positioning on the SERPs. Even changing the links, images, or even something off site, such as directory descriptions, can tell the search engines that they should re-evaluate your site to ensure that it is still relevant to the previously associated terms or perhaps has greater relevance to a new set or words of phrases.

The thought to keep in mind is that you aren’t the only one changing or optimizing text – your competitors are doing the very same with their web content. Changes made by other websites within your industry or space can indirectly influence your relevance. This is because the search engines collect and compile all that they know about all the websites and web pages they find as they crawl the Internet, and relay this information back to a search engine user based on how all of this information relates to his or her query. So as you optimize www.example.com, www.example.net and www.example.org are likely performing similar activities. Refining and targeting your content by using unique and relevant keywords, as well as keeping a level of freshness for your content and cultivating trusted and relevant inbound links to your site are just some of the ways in which you can help your website to keep its relevance in the search engine algorithms. Content remains king and relevance and usefulness are the underlying forces which ensure content will remain supremely important. Understanding how important it is to the search engines will help both your SEO efforts and your site to grow.

With Microsoft’s unsolicited bid to purchase Yahoo, how would such a merger affect the Internet, especially where search technology is concerned?

Previously, we explored how the proposed Microsoft – Yahoo merger would affect our use of the Internet. In this installment, we’ll take an historical look at the ways in which Microsoft, Yahoo, and Google have enhanced our experience on the Internet.

Before we soothsay and gaze forward to what may be with “YaSoft” or “MicroHoo” and how Google will react to either possible incarnation, let’s take a look back to where the big three search engines began.

MSN Search/Windows Live Search/Live Search
MSN Search was introduced in the fall of 1998 using search results provided from Inktomi, a search company whose technology employed the widely-used HotBot search engine. In the spring of the following year, MSN Search launched an improved version of their search engine, one which displayed listings from LookSmart, an online advertising company, blended with results from Inktomi.

Subsequent upgrades to MSN Search provided it with its own Microsoft-built search engine results, offering an index of data that was updated on a weekly or even a daily basis. MSN Search also began providing its search results to other search engine portals in an effort to better compete in the market.

In 2006, MSN Search underwent the change to Windows Live Search and then to Live Search. The changes moved beyond the offering of a search engine, index, and web crawler and expanded its use to include the ability to search for specific types of information using search tabs for the Internet, news, images, music, computer desktop, local search, and Microsoft Encarta, the digital multimedia encyclopedia published by Microsoft.

Live Search offers innovative features, among them the ability to view additional search results on the same web page (instead of needing to click through to subsequent search result pages) and the ability to dynamically adjust the amount of information displayed for each search result. It also allows users to save searches and see them updated automatically on Live.com.

Yahoo!
Before Yahoo was Yahoo, it was “Jerry’s Guide to the World Wide Web”. Yahoo was originally developed over 13 years ago as a web-based directory of sites compiled by Jerry Yang and David Filo, then students at Stanford University. The original organization of Yahoo was a hierarchical structure as opposed to a searchable index of pages. Over time, “Jerry’s Guide to the World Wide Web” developed into “Yet Another Hierarchical Officious Oracle” (Yahoo), a full-fledged portal with a search interface.

In the early 1990s, “Yahoo Search” initially referred to the Yahoo-provided search interface. Much like the early stages of MSN Search, the results presented weren’t from Yahoo per se. The actual web crawling and storage and retrieval of data were powered by Inktomi in 2001 and later, in 2004, were powered by Google.

Wanting to provide its own search engine results, Yahoo bought Inktomi in 2002. In 2003, it purchased Overture Services, Inc., which owned the AllTheWeb and AltaVista search engines. It was also in 2003 that Yahoo! Search finally evolved into its own web crawler-based search engine called “Yahoo! Slurp”. Yahoo had taken the step of combining the capabilities of the search engine companies it had purchased and put those capabilities into a single search engine. In this way, not only could Yahoo serve up its own results, but it could also sell those results to other companies. It was at this time that Yahoo’s relationship with Google was concluded (as they were now direct competitors).

One of the more recent advancements made to Yahoo! Search technology-wise is the addition of Search Assist. With Search Assist, as search terms are typed, related search terms are automatically suggested, enhancing Yahoo’s user experience.

Google
Another innovation from Stanford University came in the form of a research project called “BackRub”. The 1996 research project was initially that of Larry Page, who was soon joined by Sergey Brin while both were Ph.D. students at Stanford University. BackRub ultimately became “Google”. The name originated from a misspelling of “googol,” which refers to the number zero followed by one-hundred zeros.

The search engine originally used the Stanford website with the domain google.stanford.edu. In 1997, the domain google.com was registered, and in the following year Larry Page and Sergey Brin formally incorporated their company, Google, Inc.

By the end of 1998 Google had amassed an index of approximately 60 million pages. Although the home page was still marked as an “alpha test”, reviewers were making the argument that Google’s search results were better than those of competitors like HotBot or Excite. Google was also garnering praise for being more technologically innovative than portal sites such as Yahoo, Excite, Lycos, AOL, Go, and MSN. One of the reasons that Google found early loyalty and interest was due to its simple design and the way in which it worked. Larry Page viewed the link structure on the World Wide Web as being a huge graph. He and Sergey Brin hypothesized that a search engine which analyzed the relationships between websites would produce a better ranking of results. Their search engine took into consideration the number and the nature of web pages that link to a given page, placing the value of importance on the nature of these “backlinks”. By analyzing the output – which, for a given URL, consisted of a list of backlinks ranked by importance –Google was able to produce better results than existing techniques. Search engines at the time essentially ranked results based on how many times the search term itself appeared on a web page.

And now, back to the present: Microsoft’s $44.6 billion bid for Yahoo (which is $31.00 per share) is at a stalemate. In simple terms, Yahoo believes it is worth more than $31 a share, but it can’t force Microsoft to raise the asking price since Yahoo hasn’t (yet) been able to find a so-called “white knight”. Similarly, Microsoft doesn’t want to move away from what it views as being a fair offer.

Among the white knights that Yahoo has met with are News Corporation Online (News Corp Online) and Time Warner. Yahoo was unable to make a deal with News Corp Online. The parent company, News Corporation (News Corp), is still digesting Dow Jones and New Corp’s chairman and managing director, Rupert Murdoch, reportedly does not want to get into a bidding war with Microsoft

Time Warner, who owns AOL, may be a stronger ally than News Corp Online. Yahoo and AOL are similar in many ways: both are quite strong in webmail, instant messaging, and online display advertising, and both are legacy companies, having come to prominence during the dot-com gold rush of the late 1990s. Additionally, Google owns 5% of AOL and Yahoo and Google have a similar “friendly” ethos and openly embrace open source.

However, one week after meeting with AOL, Yahoo met with Microsoft. For the moment, the stalemate continues. With one company whose strength lies in productization, another whose strength lies in amalgamation, and another whose strength is innovation, what happens next? What could happen to the current search technology should a YaSoft/MicroHoo union occur? What, if anything, does all of this mean?

Our next installment will try to answer that very question.

With Microsoft’s unsolicited bid to purchase Yahoo, how would such a merger affect the Internet, especially where search technology is concerned?

By now, everyone – even those who aren’t addicted to the news or the Internet – has heard of Microsoft’s unsolicited bid to purchase Yahoo for a proposed $44.6 billion dollars. We’ve even touched on the subject in an earlier post. As the merger appears to be more and more likely, what exactly does the deal mean for US? How will it affect the way we use the Internet the day after the merger’s done? Or how we use it a month after? Two months after?

Before we start worrying about how we will research a new topic or how our favorite websites will fare, keep in mind that even if Yahoo runs out of options things won’t change overnight. Say that Microsoft finally succeeds in purchasing Yahoo. Given the scope of the buyout, there will be an anti-trust review in the United States. Once that has been completed, there will be another anti-trust review, one that is much lengthier and under the jurisdiction of the European Union. Should the buyout pass those two anti-trust rounds, there is the larger issue of assimilating Yahoo into the larger Microsoft whole. And that will take time.

Assimilation of Yahoo presents cultural and technological problems for Microsoft to overcome. Many of their products and technologies overlap and Microsoft will have to decide which will be combined or which will be eliminated. That in itself is a lengthy process, particularly in terms of combining products and technologies. Additionally, there are cultural concerns. Yahoo has long modeled itself as a fun-loving online company with an open-source technological model. Microsoft’s culture is more corporate and focuses on proprietary software. When the two cultures collide, there will be casualties.

Which leads us back to…how will this affect our Internet? The one we know today?

In the short term, the merger won’t have an impact outside of share prices and continued speculation in the blogosphere. The approval and assimilation processes alone should take years to complete. Add to that the fact that Google has voiced concerns regarding innovation on the Internet and has even taken the step of forming a lobbying campaign to block a Microsoft – Yahoo merger.

So, what of innovation? How different will the Internet be after such a historic union? How much will things really change once all is said and done? Stay tuned. In the next installment, we’ll take a closer look at the advancements made by Microsoft, Yahoo, and Google to see if we can determine how our use of the Internet might change post-merger.