What is a Search Engine Index?

Karen Luther - December 13, 2007

I have been throwing around the term “search engine index” without really taking the time to think about what exactly that term means and how it relates to websites. I’m pretty sure I’m not the only one and that’s a bit a shame. Once you start to think about what a search engine index encompasses you can really begin to appreciate the magnitude of work that search engines go through to try to give users the most relevant results to their queries.

When someone performs a query, the search engine does not in real time work its way through the web trying to find pages that are relevant. It has an index that it refers to in order to display a list of URLs that are related to the search query terms used. Essentially a search engine index is a database that holds a list of all the words on all the pages that the search engine has been able to find. From there a list is created of which pages those search words are found in and this data makes up the search engine index.

To generate the index, search engines use a tool referred to as spiders, also known as crawlers, to take inventory of every word found on web pages. The way spiders find your web server is by following a link to your website. It will then request that URL from your server and begin to put together a list of words found on that particular URL and place it in the search engine index. It has been reported that Google for example has over 24 billion pages indexed. I don’t know about you but I think it’s amazing that a search engine has that much reach and indexing capabilities.

In the end, the main goal of a search engine index is to optimize the speed and the methodology of finding the most relevant documents for its users. And that’s a pretty cool thing.