Forum Moderators: bakedjake

Message Too Old, No Replies

Problem with WISEnutbot (looksmart)

Zyborg is disobeying robots.txt

         

jazzguy

6:55 pm on Jul 28, 2003 (gmt 0)

10+ Year Member



For anyone that doesn't read the Search Engine Spider Identification forum, WISEnutbot/Looksmart's Zyborg bot has been identified as not obeying robots.txt and as a result, getting itself automatically banned from sites that employ trap scripts. They banned themselves on one of my sites yesterday.

The relevant threads are:
ZyBorg/1.0 violates robots.txt [webmasterworld.com]
216.88.0.0-216.91.255.255 [webmasterworld.com]

jeremy goodrich

4:34 pm on Jul 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Grub has had a problem with robots.txt for over two years and we even had Kord Campbell (here) years ago, talking about "addressing" the issue.

They (LookSmart, since they now own grub) have paid lip service to "respecting" webmasters - but have never acted on that statement. Even had a rep (here) posting about it in the last 6 months or so, but his legal team advised him to stop posting. :)

It seems that they have yet to live up to their promises, imho, I would ban the sucker if it didn't take up so much time (they don't crawl much, either - not lik Google or Alltheweb or Inkomi) so the bandwidth hit is minimal.

Though I would miss that 5 clicks I got this month if I banned 'em ;) just not sure that those 5 clicks gave me anything.