Looksmart - WiseNut Crawling Hard

To Push New Index Next Month

[larryhatch]

That's right, the Looksmart crawler paid a brief visit:
ZyBorg/1.0 [e-mail]; www.WISEnutbot.com

It didn't stay long though. It only downloaded 3 robots/txt and two 3rd-level .html pages.

Maybe they just want people to know they're still around. -Larry

[martinibuster]

There's also grub. I see grub around more often than wisenutbot.

In fact, I can't remember the last time I saw it, if ever. hehe

[jmccormac]

How very alternative. :)

Regards...jmcc

[pmkpmk]

Grub is spidering me on a very dependable schedule.

[jmccormac]

I regularly see grub clients here (I run Ireland's biggest web directory) but I rarely see any worthwhile traffic as a result of all that grub crawling. Indeed last night, one crawler that was using the grub useragent was just log spamming. At the moment it is hard to decide whether to block grub clients or leave them alone.

Regards...jmcc

[Brett_Tabke]

Saw the Wisenut crawler here too. Surprising.

[amznVibe]

Since last month:

64.242.88.10 [20/Jan/2005:20:20:30 -0600]"Mozilla/4.0 compatible ZyBorg/1.0 ([e-mail]; [WISEnutbot.com)"...]

[philaweb]

Wisenutbot has actually crawled one of my web sites in full - all pages in one go. Now I'm just looking to see whether they actually index those pages. :)

Grub does visit - about 15 hits a month, nothing much.

[donb01]

Wisenutbot usually gives me a couple hundred hits a month pretty reliably, and has for a long time now. I do have some of my higher level pages in their index.

I used to run a grub spider on one of my machines until one of my primary business sites started getting high enough in the serps to get a few gig a month in traffic, then I stopped it because I wanted the bandwidth for my visitors. I told them I wasn't crawling my own sites anymore, and removed the grub.txt file from my root level, but all I get from them is hits on robots.txt and rarely one or two hits on something else. I've never been too worried - the other reason I stopped was because they are doing all that spidering and not doing anything with the results as far as I know...

[balam]

Maybe I'm just "special" - not the first time I'd be accused of such - but Zyborg never left, as far as I can see. Sure, "he" did slow down some after Microsoft & Looksmart went their separate ways a year ago, but the crawling never stopped.

Same with ZealBot and the mysterious MARTINI... Grub, on the other hand, started looking like another Nutch or larbin, and therefore isn't allowed in the door.

Now, if someone sees Scooter - that's worthy of front page news...

[Staffa]

Are you looking for this Scooter?

01/02/05 23:17:07----66.94.232.246----Scooter/3.3

[balam]

> Scooter/3.3

Ooo! Now that's news!

What was Scooter up to? Big crawl, little crawl, just a page?

[pendanticist]

Zybor has been perusing my work for, well, longer than I can remember. I listed with Looksmart years ago.

Suppose it might be broadening it's horizons?

[Staffa]

Scooter has been coming for a while now.
Nothing much important about every week or so for a few pages at the time but it's obviously doing something.

I have looksmart/grub banned for a long time, it was forever crawling and is still trying but hardly any referals, waste of bandwidth.

[ls_wisegrub]

You are all correct. LookSmart web search engine (Wisenut) is actively crawling. And has been for the past three years. Expect to see the latest index go live in a few weeks.

Grub is also actively crawling. You may see increased activity from Grub due to the latest client - Grub v.2.5.

Finally, data collected by both crawlers is leveraged by LookSmart web search.

- LookSmart Web Search Team

[pendanticist]

Welcome to WebmasterWorld ls_wisegrub :)

Thank you for stopping by and clearing that up. Well done.

[donb01]

<Quote> "Now, if someone sees Scooter - that's worthy of front page news... " </Quote>

Scooter stops by me too, but only hits the robots.txt file. At most I get 1 hit on content. Last month I had 11 hits on robots.txt.