How do the search engines know how to find what it is that you are looking for? When you enter a query, or a series of words or a phrase into Google, MSN or Yahoo, how do they go about giving you sites that relate to your search?

Let’s try to answer this in a straightforward and somewhat simplified way. When it comes to search engine optimization, or SEO, there is usually mention of complex algorithms and predictive analytics. Let’s see if we can boil things down to a basic, real-world example.

Say you have a website, www.example.com. Typically, you would like for Internet users to find your site and to read more about your products or services. You have your content, titles, descriptions, and keywords, you’ve tackled your in-site linking and inbound linking, and you may even have a social media marketing plan in place. But how does it all come together on the back end? If you are doing everything right in terms of SEO best practices, why and how do your search engine rankings change?

At the present moment there are well over a billion pages of index-able content on the Internet. The search engines act as a way in which we can sort through all of that information, and, in turn, use that information to answer a question. The search engines collect and categorize information so they can help to answer the most basic question, that of relevance. This is how your query, the search term we mentioned earlier, relates to the information contained in the search engines’ massive databases. The question is really a matter of determining which web pages are most relevant to the terms A, B, and C.

Having over a billion pages to work through, the search engines have to manage that information in a way which ensures that less relevant information doesn’t appear at the forefront of the search engine result pages (SERPs). This isn’t part of some conspiracy – it’s a matter of trying to make the results as useful as possible, hence the focus on high relevance versus low relevance.

But how do the search engines determine what’s relevant? For this SEO blog post we’ll define “relevant” as being those web pages that have the terms which most closely match the words (keywords) the web searcher typed into the search engine.

This relevance is determined by the search engines roughly assigning a score (as in the case of Google it is PageRank), which is also how websites are listed in the SERPs. Obviously, with the vast number of websites and web pages, there is a wide variety of ways in which to measure and score relevance, some or all of which may be employed by the search engines. As these algorithms are proprietary to Google, MSN, and Yahoo, we don’t know exactly what is being used or in which way, but we do have first-hand experience, sound anecdotal evidence, as well as a wealth of research and observation to come to one conclusion when it comes to search engine relevance.

Content is king and the text is the thing. What can be considered as being the text on your website? It’s what appears in the title tags, URLs, anchor text, image alternate text, the comments (if you have a blog), the description meta tag and keywords meta tag, and in the formatted (or unformatted) visible text areas of a page. You can cultivate the relevance of pages on your site through in-site linking. The text as found on one page can be supplemented by information that is somehow associated with that page (as in a link) as well as related pages which link to that page.

Why is text so important to determining relevance? Much like a human visitor to your website uses the text on the page to figure out what the site is about and where to navigate to next, the search engines do the same. The placement of the visible text, as well as how it is emphasized and used, help the search engines understand what the pages on your site are about. Building keyword density and using keywords with prominence can help to assert the page’s relevance.

Changing the text in the pages changes the relevance. This is why there are changes to rankings and positioning on the SERPs. Even changing the links, images, or even something off site, such as directory descriptions, can tell the search engines that they should re-evaluate your site to ensure that it is still relevant to the previously associated terms or perhaps has greater relevance to a new set or words of phrases.

The thought to keep in mind is that you aren’t the only one changing or optimizing text – your competitors are doing the very same with their web content. Changes made by other websites within your industry or space can indirectly influence your relevance. This is because the search engines collect and compile all that they know about all the websites and web pages they find as they crawl the Internet, and relay this information back to a search engine user based on how all of this information relates to his or her query. So as you optimize www.example.com, www.example.net and www.example.org are likely performing similar activities. Refining and targeting your content by using unique and relevant keywords, as well as keeping a level of freshness for your content and cultivating trusted and relevant inbound links to your site are just some of the ways in which you can help your website to keep its relevance in the search engine algorithms. Content remains king and relevance and usefulness are the underlying forces which ensure content will remain supremely important. Understanding how important it is to the search engines will help both your SEO efforts and your site to grow.

With Microsoft’s unsolicited bid to purchase Yahoo, how would such a merger affect the Internet, especially where search technology is concerned?

Microsoft’s long courtship of Yahoo has now entered a serious phase, one of not-so-gentle persuasion and has somehow managed to become even more complex. It’s no longer a matter of, “Software Company seeks Web Portal for friendly merger” – events of the past week have elevated Microsoft’s chase of Yahoo into a complicated mass of what-might-be.

Since we last wrote about the major three search engine companies – Microsoft, Yahoo, and Google – volleys have been lobbed back and forth from both sides over the possible “YaSoft/MicroHoo” union. This past week alone has seen not only an increased flurry of activity, but a more intense pursuit as well.

When Microsoft’s initial $44.6 billion dollar take-over bid was first made public, Yahoo vigorously rejected it. The weeks that followed the rebuff were filled with silence from both the Yahoo and Microsoft camps. Yahoo had made it quite clear that it felt Microsoft’s offer seriously undervalued its earnings potential and, in turn, Microsoft felt that it made a fair offer.

For weeks, it seemed that was that. Neither side had any more to say on the matter. Then reports surfaced that Yahoo was talking to Time-Warner about a merger with AOL. This led to speculation that Google might leverage its 5% stake in AOL by partnering with or even acquiring AOL before Yahoo could pursue them.

Given Yahoo’s pursuit of alternate suitors (News Corp Online and Time-Warner), which may be viewed as either a way to escape Microsoft’s bid or to raise it to perhaps a $34.00 a share asking price, Microsoft issued an ultimatum: the initial $31.00 a share bid is good until April 26th. After that, the offer price will drop and there will be a proxy fight to nominate a new board of directors.

Despite this ultimatum, Yahoo continued on with business as usual, immediately responding with an announcement that Google will power 3% of paid search ads on Yahoo on a test basis. This fueled the earlier speculation that Yahoo and AOL were in serious talks and that Microsoft and News Corp Online were partnering to bid for Yahoo.

Microsoft’s reply to the Yahoo-Google search deal was to raise the issue of antitrust concerns, concerns that had been noted by Google when news of the Microsoft bid for Yahoo first surfaced. In fact, Google and Yahoo combined account for at least 80% of the search market. A full-scale outsourcing deal would place Google in control of the vast majority of paid search. However, Microsoft’s antitrust concerns are more reflective of the magnitude of a full-scale merger, one which would combine paid search, organic search, and paid advertising, and not a search outsourcing deal.

Almost as if to battle this, Yahoo made several announcements, one of which detailed a three-year deal with Major League Baseball’s MLB.com. The deal involves video distribution and related advertising, with Yahoo offering a co-branded version of the MLB.TV media player on Yahoo Sports.

In the first year of the MLB.com deal, Yahoo will offer its clickable video ad format (allowing users to click through on video ads) along with standard pre- and post-roll ads. After this initial first year period, all video ad inventory will be powered by Yahoo’s new AMP ad platform.

The new advertising management platform called AMP will be rolled out later this year. The platform is intended to offer a single interface for buying search, display, local, mobile, and video ads across all of Yahoo’s properties. Additionally, AMP will also reach across Yahoo’s premium ad partners (among them AT&T, eBay, and Viacom) as well as the sites of more than 600 publishers in the Newspaper Consortium. Consortium founders include Belo, Hearst, Lee, MediaNews Group, and Scripps.

The AMP platform is scheduled to be available to the Newspaper Consortium in the third quarter of the year and to Yahoo’s other advertising partners after that.

In addition to striking a deal with MLB.com and rolling out the AMP platform, Yahoo also announced the acquisition of IndexTools , a maker of Web analytics software for online marketing.

Of all of the recent deals struck by Yahoo, it’s this acquisition that presents a risk for them. Although IndexTools is considered by many to be superior to the free and low-cost solutions offered by other services, time can negate any benefits. For example, Yahoo could take too long to integrate IndexTools, more hiccups could appear with the Microsoft bid, and Google Analytics could offer more dramatically up-market, competitive features.

Having spent some time looking at what Yahoo has been up to let’s take a look at Microsoft’s recent activities. The flurry created with Yahoo’s meeting with Time-Warner led to an unexpected move on Microsoft’s part – the software giant reached out to News Corp Online’s MySpace. Having MySpace as an ally would fill a major gap for Microsoft, as the software giant is currently missing a social networking tie-in. This tantalizing bit of news certainly influences thoughts of what may come of YaSoft/MicroHoo.

With recent developments in hand, and assuming that the Yahoo board of directors (who met April 11th), decide to entertain Microsoft’s offer, the potential of the YaSoft/MicroHoo union fairly explodes.

Let’s start with Google and MySpace. Google has a $900 million Internet ad search deal in place with MySpace. Microsoft recently paid $240 million for a 1.6% stake and a broader marketing relationship with Facebook. An alliance of Yahoo, Microsoft, MySpace and Facebook, ostensibly to compete directly with Google, is mind-boggling.

Despite this flurry, it is still likely that Microsoft will acquire Yahoo. In the end, all of the players involved will still be focused on competing with Google. After all, Google accounts for over 50% of Internet searches. The current scramble has Microsoft, Yahoo, and AOL working to expand their respective user bases and to develop more powerful advertising mechanisms. Microsoft’s courtship of Yahoo has morphed into a strategic tit-for-tat involving MySpace and News Corp Online, and AOL and Time-Warner. The irony to this development is that Google lacks its own social networking component, yet Microsoft and Yahoo are attempting to place themselves into a better competitive position with Google by acquiring social networking.

So, should Yahoo’s board of directors accept Microsoft’s offer, whether it be at $31.00 a share or slightly higher, there are still at least 18 months of integration, streamlining and cultural clashes. This would leave Google the same 18 months to launch improvements and to move into areas where Yahoo may hold a lead (specifically, Internet banner ads).

Once these hypothetical 18 months come to a close, likely point of impact may not necessarily be search technology (Microsoft and Yahoo combined have roughly 30% of online search) but web email and instant messaging (MSN Hotmail and Yahoo Mail, and MSN Messenger/Windows Live Messenger and Yahoo Messenger). A combination of these services (of which Yahoo has the stronger presence and technology) would make YaSoft/MicroHoo the most heavily trafficked portal on the Internet.

In the short term, search technologies from MSN/Windows Live Search and Yahoo Search are likely to remain the same as it will take a significant amount of time to get through the legal hurdles, let along the logistical, technological and cultural ones. This span of time may allow for Google to test and implement further improvements to their algorithm and to build out the functionality of their Universal Search. In the short term, things should be business as usual, with the search engines delivering what we come to expect in the way of improvements with their search technologies. It’s the long term that still remains in question.

With Microsoft’s unsolicited bid to purchase Yahoo, how would such a merger affect the Internet, especially where search technology is concerned?

Previously, we explored how the proposed Microsoft – Yahoo merger would affect our use of the Internet. In this installment, we’ll take an historical look at the ways in which Microsoft, Yahoo, and Google have enhanced our experience on the Internet.

Before we soothsay and gaze forward to what may be with “YaSoft” or “MicroHoo” and how Google will react to either possible incarnation, let’s take a look back to where the big three search engines began.

MSN Search/Windows Live Search/Live Search
MSN Search was introduced in the fall of 1998 using search results provided from Inktomi, a search company whose technology employed the widely-used HotBot search engine. In the spring of the following year, MSN Search launched an improved version of their search engine, one which displayed listings from LookSmart, an online advertising company, blended with results from Inktomi.

Subsequent upgrades to MSN Search provided it with its own Microsoft-built search engine results, offering an index of data that was updated on a weekly or even a daily basis. MSN Search also began providing its search results to other search engine portals in an effort to better compete in the market.

In 2006, MSN Search underwent the change to Windows Live Search and then to Live Search. The changes moved beyond the offering of a search engine, index, and web crawler and expanded its use to include the ability to search for specific types of information using search tabs for the Internet, news, images, music, computer desktop, local search, and Microsoft Encarta, the digital multimedia encyclopedia published by Microsoft.

Live Search offers innovative features, among them the ability to view additional search results on the same web page (instead of needing to click through to subsequent search result pages) and the ability to dynamically adjust the amount of information displayed for each search result. It also allows users to save searches and see them updated automatically on Live.com.

Yahoo!
Before Yahoo was Yahoo, it was “Jerry’s Guide to the World Wide Web”. Yahoo was originally developed over 13 years ago as a web-based directory of sites compiled by Jerry Yang and David Filo, then students at Stanford University. The original organization of Yahoo was a hierarchical structure as opposed to a searchable index of pages. Over time, “Jerry’s Guide to the World Wide Web” developed into “Yet Another Hierarchical Officious Oracle” (Yahoo), a full-fledged portal with a search interface.

In the early 1990s, “Yahoo Search” initially referred to the Yahoo-provided search interface. Much like the early stages of MSN Search, the results presented weren’t from Yahoo per se. The actual web crawling and storage and retrieval of data were powered by Inktomi in 2001 and later, in 2004, were powered by Google.

Wanting to provide its own search engine results, Yahoo bought Inktomi in 2002. In 2003, it purchased Overture Services, Inc., which owned the AllTheWeb and AltaVista search engines. It was also in 2003 that Yahoo! Search finally evolved into its own web crawler-based search engine called “Yahoo! Slurp”. Yahoo had taken the step of combining the capabilities of the search engine companies it had purchased and put those capabilities into a single search engine. In this way, not only could Yahoo serve up its own results, but it could also sell those results to other companies. It was at this time that Yahoo’s relationship with Google was concluded (as they were now direct competitors).

One of the more recent advancements made to Yahoo! Search technology-wise is the addition of Search Assist. With Search Assist, as search terms are typed, related search terms are automatically suggested, enhancing Yahoo’s user experience.

Google
Another innovation from Stanford University came in the form of a research project called “BackRub”. The 1996 research project was initially that of Larry Page, who was soon joined by Sergey Brin while both were Ph.D. students at Stanford University. BackRub ultimately became “Google”. The name originated from a misspelling of “googol,” which refers to the number zero followed by one-hundred zeros.

The search engine originally used the Stanford website with the domain google.stanford.edu. In 1997, the domain google.com was registered, and in the following year Larry Page and Sergey Brin formally incorporated their company, Google, Inc.

By the end of 1998 Google had amassed an index of approximately 60 million pages. Although the home page was still marked as an “alpha test”, reviewers were making the argument that Google’s search results were better than those of competitors like HotBot or Excite. Google was also garnering praise for being more technologically innovative than portal sites such as Yahoo, Excite, Lycos, AOL, Go, and MSN. One of the reasons that Google found early loyalty and interest was due to its simple design and the way in which it worked. Larry Page viewed the link structure on the World Wide Web as being a huge graph. He and Sergey Brin hypothesized that a search engine which analyzed the relationships between websites would produce a better ranking of results. Their search engine took into consideration the number and the nature of web pages that link to a given page, placing the value of importance on the nature of these “backlinks”. By analyzing the output – which, for a given URL, consisted of a list of backlinks ranked by importance –Google was able to produce better results than existing techniques. Search engines at the time essentially ranked results based on how many times the search term itself appeared on a web page.

And now, back to the present: Microsoft’s $44.6 billion bid for Yahoo (which is $31.00 per share) is at a stalemate. In simple terms, Yahoo believes it is worth more than $31 a share, but it can’t force Microsoft to raise the asking price since Yahoo hasn’t (yet) been able to find a so-called “white knight”. Similarly, Microsoft doesn’t want to move away from what it views as being a fair offer.

Among the white knights that Yahoo has met with are News Corporation Online (News Corp Online) and Time Warner. Yahoo was unable to make a deal with News Corp Online. The parent company, News Corporation (News Corp), is still digesting Dow Jones and New Corp’s chairman and managing director, Rupert Murdoch, reportedly does not want to get into a bidding war with Microsoft

Time Warner, who owns AOL, may be a stronger ally than News Corp Online. Yahoo and AOL are similar in many ways: both are quite strong in webmail, instant messaging, and online display advertising, and both are legacy companies, having come to prominence during the dot-com gold rush of the late 1990s. Additionally, Google owns 5% of AOL and Yahoo and Google have a similar “friendly” ethos and openly embrace open source.

However, one week after meeting with AOL, Yahoo met with Microsoft. For the moment, the stalemate continues. With one company whose strength lies in productization, another whose strength lies in amalgamation, and another whose strength is innovation, what happens next? What could happen to the current search technology should a YaSoft/MicroHoo union occur? What, if anything, does all of this mean?

Our next installment will try to answer that very question.