homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

Grub crawler download

 6:07 pm on Apr 3, 2003 (gmt 0)

Looksmart are now offering Grub [webmasterworld.com] to be distributed as a screensaver, it will use you computers power to spider the web and return the results to them.

You have no control over what it spiders other than the ability to point it at your site daily to be spidered!

Looksmart/wisenut are now going to have a huge spread out base of computers to keep their results up to date.


jeremy goodrich

 6:30 pm on Apr 3, 2003 (gmt 0)

Will be interesting to see if that improves their search. So far, must say, it still needs work.

For example, my mom was looking for some site the other day - I got her to try WiseNut - and she had to scroll to the 3rd page of results to get what she wanted.

Google, it's on page #1 - not a commercial site, so it's not a case of some spammer pushing the results down or what not.

Still - distributed computing can do lots of stuff, when you think about the potential, it's pretty big - spread across lots of even DSL lines, you could get a large # of pages in a quick period of time.


 6:55 pm on Apr 3, 2003 (gmt 0)

Grub FAQ [sv-grubdb1.looksmart.com...]

Grub Download

Also gives you a visual of it's crawling as it goes, very cool.

It will be intersting to see how much better it makes their search, it has huge upside.

jeremy goodrich

 6:57 pm on Apr 3, 2003 (gmt 0)

Ya, WebmasterWorld was first on this one, few weeks ago we had a long thread on it [webmasterworld.com...]

Then followed CNET, etc al. :)

jeremy goodrich

 7:08 pm on Apr 3, 2003 (gmt 0)

Hm, that thread was on the front page here for at least a week...perhaps you should check into the WebmasterWorld RSS Feed [webmasterworld.com] :) That way you won't have to miss anything...:)


 11:00 pm on Apr 3, 2003 (gmt 0)

I have b=never been happy with grub... not as a client (I would not know about that) but as a 'bot. It is horrible, and does not respect my robots.txt. It was also gobbling up TONS of bandwidth (I have 250,000 pages or so)

It is banned on my site.



 2:29 am on Apr 4, 2003 (gmt 0)

I thought the grub bot was only allowed to crawl the front page.


 2:45 am on Apr 4, 2003 (gmt 0)

>It is banned on my site.

Mine too...until the end of my time on Earth.



 3:37 pm on Apr 4, 2003 (gmt 0)

Simply put, grub obeys robots.txt.

We will aggressively hunt down any counter-example and if there's a bug, we'll own it and fix it.

carfac/pedanticist, please sticky me with your site info and I'll personally make sure it gets taken care of. Alternatively, we have an interface in place that allows you to notify the system to recrawl your robots.txt automatically at [grub.org...] .

Also, we're in the process of rolling out some (more) scalability support over the next few days. We had hoped that people would share our enthusiasm (or should i say zeal? *grin*) for what's possible with distributed search, but we got a little surprised at how fast people are getting on board (user base has quadrupled in just the last 2 weeks). Because of these upgrades, there may be a little choppiness over the weekend...



 5:23 pm on Apr 4, 2003 (gmt 0)

What a crock! I've read all the papers on Grub going back to 2001, (including the proposed business model), and basically you think all us nerds will be so excited about another peer-to-peer open source toy we can play with that we will all gladly do all the work, save you the expense of doing it yourself and let you make money while we make sh**

Grub is all about making money that is for sure. Just not about making anything for anyone else.

You seem to think that SEO's will be so desparate for your daily spidering that we will jump at the chance to make you money while we pay a minimum of 15 cents for every click. What's in it for me? So my site gets spidered everyday. So what? Where does that put my site under what keywords?

My opinion is that deal sucks. You want my computer to make you money? Here's a novel idea, give me a cut! Throw me a bone, something besides just expecting me to love the idea of helping you for all you've done for me these past two years.

jeremy goodrich

 5:30 pm on Apr 4, 2003 (gmt 0)

Actually, that's not a bad idea. Surely there would be more people willing to 'sign up' if they had a few perks...from what I've seen, many volunteer driven web systems fail because there are no perks for volunteering.

Figure out what people who download Grub might want, give them a reason to invest their time, and you just might have something...

Otherwise, agree with shurlee - most people won't be downloading any time soon.


 6:10 pm on Apr 4, 2003 (gmt 0)

good point shurlee... here... \_/ have some more coffee. just kidding, sounds like you have been down the "grub using your computer" road before.

i agree that there should be more in it for the peeps whose computer is used, but what would you propose they do? not trying to be smart, but im curious to know your ideas about how such a model would work. that is, pay or reward on some basis the person who downloads the grub software.


 7:51 pm on Apr 4, 2003 (gmt 0)


Thanks for your post! I am glad to see your response- I assume you are with grub in some fashion (GrubGuy?)

Anyway, I did contact someone at grub when I first started having problems- well over a year ago. At that point, I was told it did NOT obey robots.txt, and they had no intention of adding that functionality.

It was suggested that I run grb myself on my site, and that way I could control it better.

This is as I recall, as I said, it was over a year ago, and I do not even remember what I ate last night!

Anyway, we ended up putting ALL my URL's into a ban at grub, so I am not crawled at all anymore. That has worked out fine. As- at that time- there was no use for the grub data (I just looked, it still does not look like you can search the grub data directly), I saw no reason for inclusion in the project.

To be blunt, the whole project reeked to me. The person I talked to seemed very secritive about how the data would be used. I have 250,000 pages on one site, and I do not want to give away the bandwidth for all those pages to a project that cannot even tell me what they are going to do with the data. I- for one- would love to SEE the grub data... to searchg it. But I cannot.... and the whole project seems rather worthless to me until one CAN search the data!

Anyway, I might possibly re-think my posiution, depending on what happens with Looksmart. I am well positioned for all my sites in Looksmart now...


jeremy goodrich

 7:58 pm on Apr 4, 2003 (gmt 0)

For the record, he works at LookSmart - and has helped answer a question or two here about things related to them before.

Ya, the Grub project - problems, history, etc. Will be interesting to see how it evolves now.


 10:29 am on Apr 6, 2003 (gmt 0)

Jeremy....for the record he is the Director of Technology at The Grub!

A person i would highly recommned users or complainants contact.

Shurlee......love to hear how you would expect The Grub to make you money?

Maybe you could post that view here too and see what Stechert says.

And finally on Stechert............i think it is commendable the Director of Technology comes on this forum to resolve issue as does the Google Guy.

Be nice if more companies were so upfront.

jeremy goodrich

 8:43 pm on Apr 6, 2003 (gmt 0)

Ya, very cute. Have you seen the Alexa rankings thread, Jillibert?

Perhaps that would explain why he's being "helpful" it's great free PR (and no, I'm NOT talking about PageRank).

That being said, fantastic that he's helping, however, it's a bit shady when the profile doens't reflect the truth, wouldn't you say?

After all, last I looked, it didn't say, in his profile that he works at LookSmart. Well, now it does - but it didn't before...:)

With GoogleGuy, it's fairly obvious where he works, thus, what his motives are in posting at WebmasterWorld.

We've all got an agenda here...some are just more transparent than others.

And in case you didn't notice, Jillibert, we've got engine reps from EVERY major search engine here, not just Google, or LookSmart. So it's really nothing new under the son.


 10:21 pm on Apr 6, 2003 (gmt 0)

NO i would not say its a bit shady.

Jeremy,its quite obvious your slant too but its not helpful to the forum for you to maintain your posture as you do.

Stechert was clearly identified early when he posted he was from LOOK and I exposed that information as he helped me on some issues.

But his previous representations were not made as an employee of LOOK but simply as a contributor to this site.

Now his role has been elevated and his is acting here now in some official capacity is to be commended.

Like i said outside Google Guy not too many others to my knowledge identify themselves as working for some of these companies as you suggest although i would be more then happy if you want to tell me who they are on this thread.

To be crtical of some companies in an onbective way is far better then how you approach it........you suggest other folk have a bias.......yours shows almost ALL the time.

After all i thought this forum was for plurality of ideas so everyone is better informed?

jeremy goodrich

 10:28 pm on Apr 6, 2003 (gmt 0)

Ack, my bad -> this thread is a bit off topic at this point, and discussing which engines post is really the topic of another thread. :)

I'm sure if you use the Community Center [webmasterworld.com] and ask about which engines have had reps post in an official capacity, a greater number of the member base would 'chime in'.

Yes, my bias is towards the open, free exchange of ideas and information -> on more search engines making their algos better.

To go back to the subject, I'm 100% certain Grub can help make a better engine - however, whether or not that happens - remains to be seen. :)

Let's try not to get personal, k?
<----- My stickymail is always open if you feel there is something further we need to discuss that is outside the scope of this thread.


 10:58 pm on Apr 6, 2003 (gmt 0)

Whether Grub works or not is way too early too tell and i am prepared to wait and see how or if it does work before i make a rush to judgement.

On the sticky mail..........you can always get me there if you need too!

But personally i like open and not closed debate.

[edited by: jeremy_goodrich at 11:08 pm (utc) on April 6, 2003]
[edit reason] see Terms of service #4 [webmasterworld.com] [/edit]

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved