When the Internet was first developed in 1971, existing as the U.S. Department of Defense’s ARPANET (Advanced Research Projects Agency Network), it was only 23 mini-computers large and spanned several universities and institutions across the United States.

In the ten years that followed, the U.S. National Science Foundation created a separate network called CSNET (Computer Science Network). This was developed for institutes without access to ARPANET. The CSNET network eventually merged with other networks to become the Internet.

Figure 1: ARPANET (then)

Figure 2: Internet (now)

The early version of the Internet was created to provide researchers with a way in which they could more easily communicate, quickly sharing theories, discoveries, and opinions. In the decades that followed the birth of the Internet, the practices of Web 1.0 and the fledgling directory and search engine technologies gave way to improvements under the Web 2.0 banner.

While the arts and technology disciplines burgeoned under Web 1.0 and came to prominence with Web 2.0 advancements, science languished. The technology that was developed by science for science left its parent field behind. But with Web 2.0 and the ever improving ways in which the search engines – and, by extension, search engine optimization (SEO) – are able to perform more relevant and functional information retrieval, science is now cautiously moving into the “Science 2.0” realm.

Much as the Internet has helped industries such as musical entertainment, hardware manufacturing, and software development realize the value of the Internet during the Web 1.0 phase, and enable such industries to monetize said value with Web 2.0, the Internet is now helping science to realize the value lying within. The ability to create a website that is easily crawled and indexed by the search engines, as well as the means by which one can create content that is identified as useful and relevant through SEO best practices, have made scientific publications open-access. This level of transparency and connectivity is driving science to challenge itself, making such traditionally opaque publications such as scientific journals and peer reviews subject to the challenge of self-sufficiency.

Web 2.0 practices, especially the rise of blogging and blogging software, now makes it easy for anyone to become a web publisher. Adopting SEO best practices, such as keyword targeting, cultivated trusted and relevant inbound links, utilizing title tags, description meta tags, and keywords meta tags, and content freshness can allow researchers to self-publish – the scientists who grew into Web 2.0 can now post ideas and data online, and optimize the information in a way that enables them to share their ideas quickly and universally. The wide-spread adoption of social media, and interactivity via social networking means that scientists who embrace Web 2.0 practices can debate and discuss on blogs, in newsgroups and forums, via mailing lists, and simply publish research documents under open-access publishing on their own website. This level of interactivity and transparency has led to a shift in the journal publication and peer review process: scientific research can now be web-published, indexed by search engines, instantly referenced and downloaded by web searchers, and the rarified practice of peer review has given way to the immediacy of peer interest.

These shifts are reflected across disciplines all over the Internet. Peer interest, or user interest, and the ability to easily search for and find information using search engines and directories, has led to virtual forests of information. These forests can only be navigated by helping the search engines determine which information is the most important. SEO practices such as writing compelling, keyword-rich content and using metadata help the search engines to be better able to serve up the best possible results for a web search. The addition of optimizing bookmarks, tags, multimedia, and other social media elements help to further refine search results. Sites such as Amazon.com, Digg.com, and Wikipedia.org have enabled critics, authors, readers, anyone, to be able to generate content and to comment upon it. This sort of “sociability” has now finally seeped its way into science.

While the age-old customs of scientific journals and peer review won’t be so readily replaced, science embracing Web 2.0 and the now adolescent child of its creation, the Internet, can only lead to faster advancements and swifter changes. After all, it was the need for improved communication that helped to bring about this modern age of Web 2.0, soon to be Web 3.0. As science becomes more and more accustomed to the way in which its child has matured, future improvements will be considerable. There are still over a billion plus pages of web content and documentation that have yet to be crawled and indexed by the search engines, information that is sitting deep within the Internet as part of the semantic web. As more and more scientists embrace the Internet and its various technologies, these deep pages, rich with content, profound with relevance, will come to the surface as science helps to push the search engines to dive deeper and to search better, truly serving up the most relevant and the most useful results… and as the Internet and the search engines develop and improve thanks to science, so will SEO. Search engine optimization continues to grow and adjust to the ever shifting Internet, learning along with the search engine algorithms and their robots and spiders. Now, decades after creating the Internet, science is back to push things along and SEO will be there, ready to adapt to the changes.

Despite the fact that Microsoft finally walked away from its takeover bid for Yahoo (a subject we’ve touched upon previously… not to worry, we’ll talk about the fallout as well), Yahoo continued with their “business as usual” approach. Yahoo Search partnered with McAfee, a leading web security company, to scan search results for dangerous websites.

While this step is new for Yahoo, it’s not new to the world of web search – Google introduced a similar protection in 2006, though the Google SafeSearch feature was aimed at screening for sites that contain explicit (i.e. adult-oriented) material, removing such sites from search results. Yahoo’s SearchScan feature takes safety to the next level.

Yahoo SearchScan uses McAfee’s SiteAdvisor technology to warn users about “potentially risky sites”. The service, which is switched “on” by default within the Yahoo Search page, warns users if they are about to click on a website that hosts viruses, spyware and spam. The warning appears as an on-screen alert, as shown here:

The service is currently available as a free beta for Yahoo Search users in the United States, Canada, the United Kingdom, France, Italy, Germany, Australia, New Zealand, and Spain

Yahoo’s SearchScan alerts users to the following security concerns:

• Browser Exploits / Hacking Risks – These types of sites can harm a user’s computer or install malware simply with a visit to the site. Such sites may use methods like browser hacks to infect a computer without any action on the user’s part.
• Dangerous Downloads – These sites may offer downloads with dangerous software like viruses or spyware embedded in them. This often comes bundled with other free downloads like screensavers, ringtones, games, or other software. SearchScan will display warnings next to search results for sites that offer such potentially dangerous software.
• Unsolicited Email – Users sharing email addresses with these types of sites may result in unsolicited or spammy emails. These sites may request users for their email address but then either send them unsolicited email, or in some cases, make users email addresses available to multiple domains to send users email. SearchScan will alert users to scanned sites that send unsolicited e-mails or inappropriately share e-mail addresses with third parties

Industry analysts at research firm IDC estimate that 67% of all computers have some form of spyware installed without a user’s knowledge . As viruses, spyware, adware, and other types of malware programs are often “hidden” inside innocuous-looking programs such as screensavers and toolbars, the Yahoo SearchScan feature helps to educate search engine users and web surfers in general to sites that may expose their computers to a security threat or may flood their inbox with spam. This enhancement to the Yahoo search engine, along with the dust settling from the Microsoft takeover falling through, can help Yahoo Search users surf the web a little more easily.

As the big three search engines (Google, MSN, and Yahoo) have been so active of late, with advancements and proposals and deals flying back and forth between them, news on other search technologies available might have been missed. We’re going to take the opportunity now to welcome Alexa and Dogpile to the search algorithm stakes.

Ask anyone involved with search engine optimization about the web traffic monitoring and rankings service Alexa (which is a division of Amazon) and you may have the answer preceded by an exasperated sigh. While the downloadable Alexa toolbar has been used by many Internet users, from the casual surfer to webmasters, the fact that the ranking site had based its statistical information from the toolbar users alone was frustrating. Some of the functionality that Alexa offered with the way it served up this data was good, but the fact that the data was, for lack of a better word, “skewed” was the cause of much frustration. It was Alexa’s heavy reliance on user data that led to the sighing and grinding of teeth.

But this is no more. Alexa recently announced that it had completely revamped its ranking protocol. Alexa said it would be moving away from releasing data based solely on users of the downloadable toolbar, to ranking sites based on info from multiple sources. The new Alexa rankings are now based on more sources of data. This provides a much better indicator of website popularity, which in turn leads to better rankings as site popularity is more accurately reflected in the new and improved algorithms. Additionally, Alexa has introduced an improved methodology to provide consistently accurate rankings for all countries.

Not to be left behind, Dogpile (owned by InfoSpace) has also made improvements. Dogpile has not only updated their updated search algorithm, but also their site design and they now offer new partner content.

Dogpile’s updates include improvements in the blending of results (the importance of blended results has really come to the fore as a result of Google’s Universal Search), the addition of tabbed search categories on the home page, the launch of a “SearchSpy” social networking widget for Netvibes or iGoogle, deep links (which are along the lines of Google site links), and integrated content from new partner agreements

While the big three search engines continue to one-up each other in terms of relevance and functionality, the smaller search engines and web ranking tools are doing the same. Alexa and Dogpile have shown that they are listening to their users and are serious about continuing to upgrade their offerings and providing their users improved features and technology. As Google, MSN, and Yahoo push one another to be bigger and better, their advancements also serve to inspire the other ranking and search technologies in the market. It will be interesting to see who will be the next entry to this stage of the race.