Forum Moderators: bakedjake
You have no control over what it spiders other than the ability to point it at your site daily to be spidered!
Looksmart/wisenut are now going to have a huge spread out base of computers to keep their results up to date.
For example, my mom was looking for some site the other day - I got her to try WiseNut - and she had to scroll to the 3rd page of results to get what she wanted.
Google, it's on page #1 - not a commercial site, so it's not a case of some spammer pushing the results down or what not.
Still - distributed computing can do lots of stuff, when you think about the potential, it's pretty big - spread across lots of even DSL lines, you could get a large # of pages in a quick period of time.
Grub Download
[grub.org...]
Also gives you a visual of it's crawling as it goes, very cool.
It will be intersting to see how much better it makes their search, it has huge upside.
Then followed CNET, etc al. :)
It is banned on my site.
dave
Mine too...until the end of my time on Earth.
Pendanticist.
We will aggressively hunt down any counter-example and if there's a bug, we'll own it and fix it.
carfac/pedanticist, please sticky me with your site info and I'll personally make sure it gets taken care of. Alternatively, we have an interface in place that allows you to notify the system to recrawl your robots.txt automatically at [grub.org...] .
Also, we're in the process of rolling out some (more) scalability support over the next few days. We had hoped that people would share our enthusiasm (or should i say zeal? *grin*) for what's possible with distributed search, but we got a little surprised at how fast people are getting on board (user base has quadrupled in just the last 2 weeks). Because of these upgrades, there may be a little choppiness over the weekend...
Cheers!
Grub is all about making money that is for sure. Just not about making anything for anyone else.
You seem to think that SEO's will be so desparate for your daily spidering that we will jump at the chance to make you money while we pay a minimum of 15 cents for every click. What's in it for me? So my site gets spidered everyday. So what? Where does that put my site under what keywords?
My opinion is that deal sucks. You want my computer to make you money? Here's a novel idea, give me a cut! Throw me a bone, something besides just expecting me to love the idea of helping you for all you've done for me these past two years.
Figure out what people who download Grub might want, give them a reason to invest their time, and you just might have something...
Otherwise, agree with shurlee - most people won't be downloading any time soon.
i agree that there should be more in it for the peeps whose computer is used, but what would you propose they do? not trying to be smart, but im curious to know your ideas about how such a model would work. that is, pay or reward on some basis the person who downloads the grub software.
Thanks for your post! I am glad to see your response- I assume you are with grub in some fashion (GrubGuy?)
Anyway, I did contact someone at grub when I first started having problems- well over a year ago. At that point, I was told it did NOT obey robots.txt, and they had no intention of adding that functionality.
It was suggested that I run grb myself on my site, and that way I could control it better.
This is as I recall, as I said, it was over a year ago, and I do not even remember what I ate last night!
Anyway, we ended up putting ALL my URL's into a ban at grub, so I am not crawled at all anymore. That has worked out fine. As- at that time- there was no use for the grub data (I just looked, it still does not look like you can search the grub data directly), I saw no reason for inclusion in the project.
To be blunt, the whole project reeked to me. The person I talked to seemed very secritive about how the data would be used. I have 250,000 pages on one site, and I do not want to give away the bandwidth for all those pages to a project that cannot even tell me what they are going to do with the data. I- for one- would love to SEE the grub data... to searchg it. But I cannot.... and the whole project seems rather worthless to me until one CAN search the data!
Anyway, I might possibly re-think my posiution, depending on what happens with Looksmart. I am well positioned for all my sites in Looksmart now...
dave
Ya, the Grub project - problems, history, etc. Will be interesting to see how it evolves now.
A person i would highly recommned users or complainants contact.
Shurlee......love to hear how you would expect The Grub to make you money?
Maybe you could post that view here too and see what Stechert says.
And finally on Stechert............i think it is commendable the Director of Technology comes on this forum to resolve issue as does the Google Guy.
Be nice if more companies were so upfront.
Perhaps that would explain why he's being "helpful" it's great free PR (and no, I'm NOT talking about PageRank).
That being said, fantastic that he's helping, however, it's a bit shady when the profile doens't reflect the truth, wouldn't you say?
After all, last I looked, it didn't say, in his profile that he works at LookSmart. Well, now it does - but it didn't before...:)
With GoogleGuy, it's fairly obvious where he works, thus, what his motives are in posting at WebmasterWorld.
We've all got an agenda here...some are just more transparent than others.
And in case you didn't notice, Jillibert, we've got engine reps from EVERY major search engine here, not just Google, or LookSmart. So it's really nothing new under the son.
Jeremy,its quite obvious your slant too but its not helpful to the forum for you to maintain your posture as you do.
Stechert was clearly identified early when he posted he was from LOOK and I exposed that information as he helped me on some issues.
But his previous representations were not made as an employee of LOOK but simply as a contributor to this site.
Now his role has been elevated and his is acting here now in some official capacity is to be commended.
Like i said outside Google Guy not too many others to my knowledge identify themselves as working for some of these companies as you suggest although i would be more then happy if you want to tell me who they are on this thread.
To be crtical of some companies in an onbective way is far better then how you approach it........you suggest other folk have a bias.......yours shows almost ALL the time.
After all i thought this forum was for plurality of ideas so everyone is better informed?
I'm sure if you use the Community Center [webmasterworld.com] and ask about which engines have had reps post in an official capacity, a greater number of the member base would 'chime in'.
Yes, my bias is towards the open, free exchange of ideas and information -> on more search engines making their algos better.
To go back to the subject, I'm 100% certain Grub can help make a better engine - however, whether or not that happens - remains to be seen. :)
Let's try not to get personal, k?
<----- My stickymail is always open if you feel there is something further we need to discuss that is outside the scope of this thread.
On the sticky mail..........you can always get me there if you need too!
But personally i like open and not closed debate.
[edited by: jeremy_goodrich at 11:08 pm (utc) on April 6, 2003]
[edit reason] see Terms of service #4 [webmasterworld.com] [/edit]
