Forum Moderators: open

Message Too Old, No Replies

Article on Google's Distributed Toolbar Vs. Looksmart's Grub.org

No plans for Google to use distributed crawling

         

born2drv

4:48 am on Jun 3, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I stumbled across this and thought some of you might find it interesting:
[business2.com...]

"I talked with Google's director of search quality, Peter Norvig, about Grub's distributed, or "grid," model, and he reminded me that Google has experience with grid computing. Web surfers can use the Google toolbar to run Google Compute, which allows their computers to process scientific data when idle, much in the manner of the popular SETI@home screensaver. Norvig says Google is learning about distributed computing from this experiment; he also notes that the company's acquisition of Blogger from Pyra Labs is teaching the company a lot about crawling these new, highly dynamic sites. But for now he won't discuss any plans to add Grub-like distributed search to Google's operation, insisting that Google doesn't need other users' resources to create an excellent Web index."

starec

7:47 am on Jun 3, 2003 (gmt 0)

10+ Year Member



Grub really gives a bad reputation to distributed indexing. It does not respect robots.txt, aggressively eats bandwidth and fully deserves '403 Forbidden' on any 10+ pages website.

If Google ever decides to try this path, I hope they do it their, Google way...

percentages

7:57 am on Jun 3, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



>Norvig says Google is learning about distributed computing

In my humble experience Google is a classic example of distributed computing today. I can't think of any commercial company that has implemented distributed computing to greater effect!

There are always "brain waves" that can come up with a better theoretical model, but how many of those can really make them happen to the extent Google already has?

We are talking about the difference between theory and practice here. Those that like the theoretical concepts will always win the philosophical argument, those that like practical implementation will quote Google!

outrun

8:03 am on Jun 3, 2003 (gmt 0)

10+ Year Member



I got grub primarily to to spider my sites, I participated in remote crawling and saw how much porn and spam I was helping to index, maybe they have algos to weed them out later but I didnt want to assist in indexing porn and and spam.

regards,
Mark