Scrubby 2.2 - Crawler, Spider, and User Agent ID forum at WebmasterWorld - WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

Scrubby 2.2

Is this a new spider?

Nick_W

2:59 pm on Sep 22, 2002 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Not seen it before and a site search turned up nothing. www.scrubtheweb.com

Anyone have any info on this?

Nick

korkus2000

4:37 pm on Sep 22, 2002 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Small engine. Doesn't really hit your site that much. I don't think it is even worth banning. Seems to respect robots.txt.

here is a thread:
[webmasterworld.com...]