Forum Moderators: phranque

Message Too Old, No Replies

Robot running amok on my site

         

SlowMove

1:16 am on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The following agent is grabbing the same page over and over:
W3CLinkChecker/Revision: libwww-perl/5.69

I've never really needed to use .htaccess to block. I'm really not up to speed on this. I tried the following, and it didn't work. Please, tell me what I'm doing wrong.

RewriteCond %{HTTP_USER_AGENT} W3CLinkChecker/Revision: libwww-perl/5.69 [NC,OR]

jdMorgan

3:33 am on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



You'll need to escape the special characters, in this case, spaces and periods. Add the start and end anchors if this is the full user-agent string -- it'll speed up the compare slightly.

RewriteCond %{HTTP_USER_AGENT} ^W3CLinkChecker/Revision:\ libwww-perl/5\.69$ [NC,OR]

Jim

SlowMove

3:55 am on Apr 2, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thank you very much.