What resources to index the whole web?

[SlyOldDog]

I was thinking about indexing the whole web (not graphics, just text).

What resources do you think I will need? A cigarette paper calculation for my estimate:

If there are 8 billion pages with an average size of 30kb (without graphics) that is 240 terabytes.

So I would need 240 terabytes/100 gb = 2400 hard disks.

If I crawl on a 10mb line that gives me 1,2500,000 bytes per second. To crawl 240 terabytes at that rate would take 2000 days (6 years), so I guess you need a 100Mb/s connection, making 6 months to crawl the web.

how many computers would I need?

[grandpa]

Its rumored that a 1 terabyte drive is close at hand. So you can cut some of that hardware significantly - just 240 hard disks.

[Chndru]

I think a slightly difficult question is, how will you know, when you have crawled all their is to crawl? considering the dynamics of the web, that poses a bigger challenge, i think,

[lawman]

You'll know when you reach THIS [mythologic.net] page. :)

[cmatcme]

Where's the beginning of the internet then?

cmatcme

[edited by: me at now (my time zone) on today ]
[edit reason: corrected grammar ]

[lawman]

I haven't found it yet.

[rocknbil]

I do believe this was the beginning [slac.stanford.edu].

For this project all you would need can be found here [onzin.nl].

[cmatcme]

Found the download all the internet page but it closed the window.

cmatcme

[SlyOldDog]

Thanks for the very interesting comments on the history of the net guys :)

Any more ideas on the hardware?

[BlobFisk]

You're taking on an ambitious and very large project there! Depending what you want to do with the data you should look at what Google use!

Not only will you need storage, but you'll need big beefy machines on quick connections to run your spider software, index the pages (to a DB?) and follow each and every link...

Good luck!

[rocknbil]

cmatcme - that's by design. :-) It does the "drive a" thing and closes when you click OK to further frustrate the visitor.

Sorry for having fun with your thread Sly - in reality, it's far beyond most of us. You really need a room full of computers connected in an array, and sometimes even different locations. My sys admin gave me a 10-minute session one day, that's how Google accesses their data so quickly.

[Milamber]

So I would need 240 terabytes/100 gb = 2400 hard disks.

Just wait 7 years or so, then you should only need 1 hard drive.
[news.bbc.co.uk...]

[gamiziuk]

Gee, you will need BOTH Cable and DSL to get working on this project...
;)

[httpwebwitch]

if you're only downloading text you could save a lot of disk space by removing all the punctuation from the internet as you download it i recommend you should get a few dozen really big raid arrays not pc hard drives you could try kazaa too

[Leosghost]

Would you want it with or without the adsense on the pages?

[SlyOldDog]

Heh, I just need to store the html links to spider onto the next pages and I need the contact pages. Maybe it's easier than I thought.