Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: bakedjake
I used to run a grub spider on one of my machines until one of my primary business sites started getting high enough in the serps to get a few gig a month in traffic, then I stopped it because I wanted the bandwidth for my visitors. I told them I wasn't crawling my own sites anymore, and removed the grub.txt file from my root level, but all I get from them is hits on robots.txt and rarely one or two hits on something else. I've never been too worried - the other reason I stopped was because they are doing all that spidering and not doing anything with the results as far as I know...
Same with ZealBot and the mysterious MARTINI... Grub, on the other hand, started looking like another Nutch or larbin, and therefore isn't allowed in the door.
Now, if someone sees Scooter - that's worthy of front page news...
Grub is also actively crawling. You may see increased activity from Grub due to the latest client - Grub v.2.5.
Finally, data collected by both crawlers is leveraged by LookSmart web search.
- LookSmart Web Search Team