LS grub bot moving in?

Spent 24 hours straight, then 18+ the next day.

[juniperwasting]

Looking through my logs from the weekend this morning, I spied a visitor. More like a house guest. This grub bot from Looksmart in a 48 hour period, roamed one of my sites for 42 hours.

I cannot think of a reason for this, it has never happened before.
We recently canceled our agreement with LS, due to the frighteningly low ROI.

Anybody got a clue as to why this bot would move in on me like this, and should I just ban it?

[jeremy goodrich]

It's their distributed spider, if you poke around, there are lots of people that have banned it (do you get traffic from WiseNut?)

If you don't, then you might add it to the 'ban' list, as until we know that they are driving traffic, you might not want them pounding your server. Distributed spidering causes a bigger load than regular spidering, as your site needs to be crawled multiple times for accuracy.

And this is before any listing in a database of theirs...then, if they 'refresh' regularly, they'll need to do the same duplicate checking / crawling...

[juniperwasting]

Wisenut is not bringing in any real traffic what so ever. I think I will ban it. With the loss of MSN via LS I can't think of a reason to let them pound my server like this. Thanks.

[edited for FFS (fat finger syndrome)]

[stechert]

Hi there,

Just checked on your site and see that there are about 60 active URLs in the Grub DB for your domain. Grub won't schedule the URLs more than once a day - please let me know if you saw different.

Also, I notice that you have a spurious CRLF between your "User-agent: *" line and the "Disallow" lines in your robots.txt file. I think we catch and handle this case, but it's probably better not to rely on that (strictly speaking, robots.txt "records" are separated by blank lines).

In any case, you can ban via user-agent or add to robots.txt. We'll keep working on the traffic and hope to hear from your servers again someday...

Cheers,
Andre

[juniperwasting]

Andre,

I appreciate your taking the time to look things over. If I was really getting traffic from Wisenut or LS for that matter I would be more forgiving of the grub hitting me like this. I think that I will just keep my eye on the matter and make a banning decision later.

~Will

[martinibuster]

Wisenut has to refresh their database and distribute their serps cheaply or FREE, if LS wants to keep people plugging away on the Grub client, and to prevent more webmasters from banning Grub altogether.

Hey, we give stuff away for free in order to get, LS should give away their Wisenut serps on the condition that LS PPC is shown for the first 3-5 results. That would give folks an incentive to operate the Grub Client. Wider search distribution will lift the PPC revenue.

Geez, what's up with LS, can't they get a marketing clue? They're big on "Charge charge charge" but weak on properly marketing their assets.

[jeremy goodrich]

>properly marketing their assets

Seems that "free XML syndication" is a hot deal for both SearchHippo [searchippo.com] as well as Gigablast [gigablast.com].

I think you may be on to something there, martinibuster. ;) A directory / most people won't browse to the granular level they're lookin' for...but a search engine offers the ease of a modern day oracle of delphi / simply ask & get your answer - instantly.

LS could also learn a *lot* by considering the whole "gigaboost" idea - a great one if I've ever seen one. SearchHippo does the same, afaik.

Till they get their marketing up to speed with their lightening fast & courteous tech staffers, I'm a bit pessimistic on what'll happen with their web crawling ambitions.