How Do Indexes Work?

An index – an index database – is made by programming ‘spider’ or ‘robot’ software to automatically visit web pages and send back information. The information from a web page is processed and stored along with billions of others.

A complete index is a database containing a copy or description of every web page and other resource out there.

When people want to search, they go to a web page (such as at Google.com) and they fill in search terms. The search terms they put in will be words that are likely to be found on an appropriate document.

When they press the Submit button, a ‘search engine’ goes through the database pulling out sites it finds that match the search terms.

Results are returned to the user in the form of links and descriptions on a web page. The user clicks on a link to visit the actual web page or resource.

An ‘ideal’ index would have information on every resource available at any moment and it would be able to give it up on request. An ‘ideal’ search engine would be able to find exactly what you want – and as much of it as you need – and give it to you immediately.

The quality of an index can be measured in various ways. The quality of a search engine can also be measured – and standards can be established. What is an ideal index? What is an ideal search engine? What is an ideal spider?