Can search engines keep a copy of WWW if it keeps growing at this rate

A question related to the space taken by the Internet

[KFish]

I have a very amazing question.

The Internet is growing at a very rapid rate. Probably many thousands if not millions of pages are added everyday.

The search engines have to index them and keep a copy in their databases. They almost index every page generated online, irrespective of whether they will ever show up in search results.

It means everyday the search engines database is also growing massively. In other words the search engines are actually web-hosts of the whole World Wide Web.

They also have to keep many copies of their database. It means hosting the whole World Wide Web a multiple number of times.

I hope you are now getting the point what I am trying to say. ALL this is taking a lot of space... a hell lot of space.

My questions are:

1. How far is this possible? In terms of web-space, is there any saturation point where the Internet will stop growing or even decline. What can be that point?

2. Will this impact the speed of search engine algorithms to extract right pages for the queries generated? We all know there are more than 100 factors search engines take into consideration before they declare results. Is there any formula that this algorithm is independent of the size of the database?

3. Let’s talk about physical space required. Agreed that the medium used to store data is shrinking and may be manageable for a few years from now, but what after 10 years or after 100 years?

It will be great to listen to what you guys have to say.

[Receptional]

That's a great question. I think the problems are complicated by the inextricable move twoards content being rich media - audio, images and video - all of which are massively more demanding on space.

The challenge must be huge, but I think the the search engines intend to try and as I understand it this is the biggest thing to provide a barrier to entry for companies other than the big three. They don't believe anyone else can afford the investment to catch up.

They may be right. But certainly Microsoft and Google at least both feel that they can take everything the world can throw at it and more... although I don't know if they have worked out how to actually meet the demand.

[tomda]

Go for local content in local directories... Local is better than global, especially if global turns to be too global and messy.

There are few websites that allow you to backup the all Internet so thata you burn it on your new 3-terabytes CD-ROM at 10,000 USD each :)