How Big is the Internet?
There are estimated to be over 5 billion web pages out there, with thousands more being added daily. Currently, individual indexes (or ‘indices’) manage to cover less than 50% of the Internet at best, with the rest remaining unseen and unavailable to most users. The better indices may visit a site on average four times per year, and might be considerably out of date at any given time.
Large commercial indexes like AltaVista and Google utilize thousands of computers hosting thousands of gigabytes of data and handling thousands of searches every second. Despite the considerable resources devoted to the problem, it is difficult to keep up with the day to day changes.
To completely cover the Internet would take a huge corporation with immense power and control… or would it?
The Internet is by design distributed.
Why can’t it just index itself?