Using the Web: Search Engines

Elmhurst College
A. C. Buehler Library

How Big is the World Wide Web?

According to the Chicago Tribune:¹

As of mid-1999, there were approximately 800 million publicly indexable web pages (about 15 terabytes).

The most comprehensive search engine, Northern Light, covers only about 16% of publicly indexable web pages. Another popular search engine, Hotbot, covers about 11%.

(This definition is a direct quote from whatis.com.)²

On the Internet, a search engine has three parts:

A spider (also called a "crawler" or a "bot") that goes to every page or representative

pages on every Web site that wants to be searchable and reads it, using hypertext links

on each page to discover and read a site's other pages.

A program that creates a huge index (sometimes called a "catalog") from the pages that have been read.

A program that receives your search request, compares it to the entries in the index, and returns results to you. An alternative to using a search engine is to explore a structured directory of topics. Yahoo, which also lets you use its search engine, is the most widely-used directory on the Web. A number of Web portal sites offer both the search engine and directory approaches to finding information.