homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

Looksmart - WiseNut Crawling Hard
To Push New Index Next Month

 6:53 am on Jan 31, 2005 (gmt 0)

That's right, the Looksmart crawler paid a brief visit:
ZyBorg/1.0 zyborg@looksmart.net; www.WISEnutbot.com

It didn't stay long though. It only downloaded 3 robots/txt and two 3rd-level .html pages.

Maybe they just want people to know they're still around. -Larry



 7:16 am on Jan 31, 2005 (gmt 0)

There's also grub. I see grub around more often than wisenutbot.

In fact, I can't remember the last time I saw it, if ever. hehe


 11:33 am on Jan 31, 2005 (gmt 0)

How very alternative. :)



 11:36 am on Jan 31, 2005 (gmt 0)

Grub is spidering me on a very dependable schedule.


 11:48 am on Jan 31, 2005 (gmt 0)

I regularly see grub clients here (I run Ireland's biggest web directory) but I rarely see any worthwhile traffic as a result of all that grub crawling. Indeed last night, one crawler that was using the grub useragent was just log spamming. At the moment it is hard to decide whether to block grub clients or leave them alone.



 11:07 am on Feb 3, 2005 (gmt 0)

Saw the Wisenut crawler here too. Surprising.


 11:19 am on Feb 3, 2005 (gmt 0)

Since last month: [20/Jan/2005:20:20:30 -0600]"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; [WISEnutbot.com)"...]


 11:43 am on Feb 3, 2005 (gmt 0)

Wisenutbot has actually crawled one of my web sites in full - all pages in one go. Now I'm just looking to see whether they actually index those pages. :)

Grub does visit - about 15 hits a month, nothing much.


 2:07 pm on Feb 3, 2005 (gmt 0)

Wisenutbot usually gives me a couple hundred hits a month pretty reliably, and has for a long time now. I do have some of my higher level pages in their index.

I used to run a grub spider on one of my machines until one of my primary business sites started getting high enough in the serps to get a few gig a month in traffic, then I stopped it because I wanted the bandwidth for my visitors. I told them I wasn't crawling my own sites anymore, and removed the grub.txt file from my root level, but all I get from them is hits on robots.txt and rarely one or two hits on something else. I've never been too worried - the other reason I stopped was because they are doing all that spidering and not doing anything with the results as far as I know...


 3:02 pm on Feb 3, 2005 (gmt 0)

Maybe I'm just "special" - not the first time I'd be accused of such - but Zyborg never left, as far as I can see. Sure, "he" did slow down some after Microsoft & Looksmart went their separate ways a year ago, but the crawling never stopped.

Same with ZealBot and the mysterious MARTINI... Grub, on the other hand, started looking like another Nutch or larbin, and therefore isn't allowed in the door.

Now, if someone sees Scooter - that's worthy of front page news...


 6:49 pm on Feb 3, 2005 (gmt 0)

Are you looking for this Scooter?

01/02/05 23:17:07----


 7:01 pm on Feb 3, 2005 (gmt 0)

> Scooter/3.3

Ooo! Now that's news!

What was Scooter up to? Big crawl, little crawl, just a page?


 7:01 pm on Feb 3, 2005 (gmt 0)

Zybor has been perusing my work for, well, longer than I can remember. I listed with Looksmart years ago.

Suppose it might be broadening it's horizons?


 7:36 pm on Feb 3, 2005 (gmt 0)

Scooter has been coming for a while now.
Nothing much important about every week or so for a few pages at the time but it's obviously doing something.

I have looksmart/grub banned for a long time, it was forever crawling and is still trying but hardly any referals, waste of bandwidth.


 7:50 pm on Feb 3, 2005 (gmt 0)

You are all correct. LookSmart web search engine (Wisenut) is actively crawling. And has been for the past three years. Expect to see the latest index go live in a few weeks.

Grub is also actively crawling. You may see increased activity from Grub due to the latest client - Grub v.2.5.

Finally, data collected by both crawlers is leveraged by LookSmart web search.

- LookSmart Web Search Team


 7:54 pm on Feb 3, 2005 (gmt 0)

Welcome to WebmasterWorld ls_wisegrub :)

Thank you for stopping by and clearing that up. Well done.


 4:37 am on Feb 4, 2005 (gmt 0)

<Quote> "Now, if someone sees Scooter - that's worthy of front page news... " </Quote>

Scooter stops by me too, but only hits the robots.txt file. At most I get 1 hit on content. Last month I had 11 hits on robots.txt.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved