Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: bakedjake
It didn't stay long though. It only downloaded 3 robots/txt and two 3rd-level .html pages.
Maybe they just want people to know they're still around. -Larry
I used to run a grub spider on one of my machines until one of my primary business sites started getting high enough in the serps to get a few gig a month in traffic, then I stopped it because I wanted the bandwidth for my visitors. I told them I wasn't crawling my own sites anymore, and removed the grub.txt file from my root level, but all I get from them is hits on robots.txt and rarely one or two hits on something else. I've never been too worried - the other reason I stopped was because they are doing all that spidering and not doing anything with the results as far as I know...
Same with ZealBot and the mysterious MARTINI... Grub, on the other hand, started looking like another Nutch or larbin, and therefore isn't allowed in the door.
Now, if someone sees Scooter - that's worthy of front page news...
I have looksmart/grub banned for a long time, it was forever crawling and is still trying but hardly any referals, waste of bandwidth.
Grub is also actively crawling. You may see increased activity from Grub due to the latest client - Grub v.2.5.
Finally, data collected by both crawlers is leveraged by LookSmart web search.
- LookSmart Web Search Team