homepage Welcome to WebmasterWorld Guest from 54.237.213.31
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Alternative Search Engines
Forum Library, Charter, Moderators: bakedjake

Alternative Search Engines Forum

    
Looksmart - WiseNut Crawling Hard
To Push New Index Next Month
larryhatch

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 6:53 am on Jan 31, 2005 (gmt 0)

That's right, the Looksmart crawler paid a brief visit:
ZyBorg/1.0 zyborg@looksmart.net; www.WISEnutbot.com

It didn't stay long though. It only downloaded 3 robots/txt and two 3rd-level .html pages.

Maybe they just want people to know they're still around. -Larry

 

martinibuster

WebmasterWorld Administrator martinibuster us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 1607 posted 7:16 am on Jan 31, 2005 (gmt 0)

There's also grub. I see grub around more often than wisenutbot.

In fact, I can't remember the last time I saw it, if ever. hehe

jmccormac

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



 
Msg#: 1607 posted 11:33 am on Jan 31, 2005 (gmt 0)

How very alternative. :)

Regards...jmcc

pmkpmk

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 11:36 am on Jan 31, 2005 (gmt 0)

Grub is spidering me on a very dependable schedule.

jmccormac

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



 
Msg#: 1607 posted 11:48 am on Jan 31, 2005 (gmt 0)

I regularly see grub clients here (I run Ireland's biggest web directory) but I rarely see any worthwhile traffic as a result of all that grub crawling. Indeed last night, one crawler that was using the grub useragent was just log spamming. At the moment it is hard to decide whether to block grub clients or leave them alone.

Regards...jmcc

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 1607 posted 11:07 am on Feb 3, 2005 (gmt 0)

Saw the Wisenut crawler here too. Surprising.

amznVibe

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 11:19 am on Feb 3, 2005 (gmt 0)

Since last month:
64.242.88.10 [20/Jan/2005:20:20:30 -0600]"Mozilla/4.0 compatible ZyBorg/1.0 (wn-12.zyborg@looksmart.net; [WISEnutbot.com)"...]

philaweb

5+ Year Member



 
Msg#: 1607 posted 11:43 am on Feb 3, 2005 (gmt 0)

Wisenutbot has actually crawled one of my web sites in full - all pages in one go. Now I'm just looking to see whether they actually index those pages. :)

Grub does visit - about 15 hits a month, nothing much.

donb01

10+ Year Member



 
Msg#: 1607 posted 2:07 pm on Feb 3, 2005 (gmt 0)

Wisenutbot usually gives me a couple hundred hits a month pretty reliably, and has for a long time now. I do have some of my higher level pages in their index.

I used to run a grub spider on one of my machines until one of my primary business sites started getting high enough in the serps to get a few gig a month in traffic, then I stopped it because I wanted the bandwidth for my visitors. I told them I wasn't crawling my own sites anymore, and removed the grub.txt file from my root level, but all I get from them is hits on robots.txt and rarely one or two hits on something else. I've never been too worried - the other reason I stopped was because they are doing all that spidering and not doing anything with the results as far as I know...

balam

10+ Year Member



 
Msg#: 1607 posted 3:02 pm on Feb 3, 2005 (gmt 0)

Maybe I'm just "special" - not the first time I'd be accused of such - but Zyborg never left, as far as I can see. Sure, "he" did slow down some after Microsoft & Looksmart went their separate ways a year ago, but the crawling never stopped.

Same with ZealBot and the mysterious MARTINI... Grub, on the other hand, started looking like another Nutch or larbin, and therefore isn't allowed in the door.

Now, if someone sees Scooter - that's worthy of front page news...

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 6:49 pm on Feb 3, 2005 (gmt 0)

Are you looking for this Scooter?

01/02/05 23:17:07----66.94.232.246----Scooter/3.3

balam

10+ Year Member



 
Msg#: 1607 posted 7:01 pm on Feb 3, 2005 (gmt 0)

> Scooter/3.3

Ooo! Now that's news!

What was Scooter up to? Big crawl, little crawl, just a page?

pendanticist

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 7:01 pm on Feb 3, 2005 (gmt 0)

Zybor has been perusing my work for, well, longer than I can remember. I listed with Looksmart years ago.

Suppose it might be broadening it's horizons?

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 7:36 pm on Feb 3, 2005 (gmt 0)

Scooter has been coming for a while now.
Nothing much important about every week or so for a few pages at the time but it's obviously doing something.

I have looksmart/grub banned for a long time, it was forever crawling and is still trying but hardly any referals, waste of bandwidth.

ls_wisegrub

10+ Year Member



 
Msg#: 1607 posted 7:50 pm on Feb 3, 2005 (gmt 0)

You are all correct. LookSmart web search engine (Wisenut) is actively crawling. And has been for the past three years. Expect to see the latest index go live in a few weeks.

Grub is also actively crawling. You may see increased activity from Grub due to the latest client - Grub v.2.5.

Finally, data collected by both crawlers is leveraged by LookSmart web search.

- LookSmart Web Search Team

pendanticist

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1607 posted 7:54 pm on Feb 3, 2005 (gmt 0)

Welcome to WebmasterWorld ls_wisegrub :)

Thank you for stopping by and clearing that up. Well done.

donb01

10+ Year Member



 
Msg#: 1607 posted 4:37 am on Feb 4, 2005 (gmt 0)

<Quote> "Now, if someone sees Scooter - that's worthy of front page news... " </Quote>

Scooter stops by me too, but only hits the robots.txt file. At most I get 1 hit on content. Last month I had 11 hits on robots.txt.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Alternative Search Engines
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved