Cost to Crawl

estimate from a PHD thesis

[old_expat]

"An estimation for the cost of an entire crawl of the World Wide Web is about US $1.5 Million [CCHM04],
considering just the network bandwidth necessary to download the pages .."

[Lord Majestic]

Its near zero if you use the right approach - using distibuted crawling that takes advantage of idle broadband connections that already paid for via fix fee.

[old_expat]

Hello LM, can you expand on this a bit. I'm not very technical.

[inbound]

Distributed crawling is not as cheap as you think it may be.

Costs include (but are not resticted to):

bandwidth involved in getting the information to a central database
servers that the DB sits on
electricity that powers the DB servers
somewhere for the servers to 'live' safely
backup costs
staffing/programming, you can't shove the whole web into a flat file database - you need people with really good experience of dealing with more data than you can imagine
also
cost of making people aware of the project
cost of supporting people who opt in

$1.5 Million to crawl the whole web (or at least 50 Billion pages of it), If anyone can do it for that (in a reasonable time frame) I know a company who would probably bite your hand off.

[old_expat]

The guy (thesis) was talking about bandwidth only, I believe.

[runarb]

The thesis is locaated her: [dcc.uchile.cl...]

[Lord Majestic]

old_expat: check Wikipedia for "distributed crawling" :)

If anyone can do it for that (in a reasonable time frame) I know a company who would probably bite your hand off.

We crawl 50 mln per day and its scalable linearly, ie 10 times people join and we will get to 500 mln per day, that's 3 months to crawl 50 bln pages.

Of course the challenges that you mentioned are all true but our work pretty much proven that it can be done without millions needed to be invested. Certainly takes time and effort but its do-able - much harder task to actually have good ranking that is competitive with Top tier search engines.

I can't say more because of "self-promotion" rules here but those who seek will find, just like those who dare win ;)

[old_expat]

Runard - thanks for adding that link. I didn't do it because I'm never sure of the TOS re: linking, here on WW

[inbound]

much harder task to actually have good ranking that is competitive with Top tier search engines.

Indeed: a little look at your project confirms the difficulty of ranking pages well.

What you've achieved is impressive, I applaud people who have the vision and determination to do things on the scale you are attempting.

However, the real value of any search engine is in the ability to remove spam/poor pages and return the most relevant ones for searches. I'm sure you are learning a great deal about how tricky that must be (I can't profess to know much about that).

It looks as though I've got another interesting site to visit on a regular basis.

Best of luck with it.

[Lord Majestic]

Thanks inbound. It is true that its the ranking that's the most difficult thing rather than getting data in the first place. I am positive however that its not impossible to achieve with fairly limited resources, though being able to distribute workload allows to actually narrow the gap. Its certainly very hard, but nobody said it will be easy and for me victory is sweeter if it was hard fought for :)

[JamesR3]

Lord Majestic, care to PM me or elaborate on what you are doing? Might be some opportunities for collaboration.