Welcome to WebmasterWorld Guest from 54.158.227.99

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Should I block bot*

Wondering about unknown robots

     
2:48 pm on Jul 30, 2010 (gmt 0)

New User

5+ Year Member

joined:Nov 11, 2008
posts: 12
votes: 0



Hello,

Bandwidth on one of our sites was severely messed with on the 28th of this month. The cpanel only tracks daily access logs and didnt archive them(it does now), using aw stats I found our bot traffic to be as follows:

Unknown robot (identified by 'bot*') 91541+417 4.78 GB 28 Jul 2010 - 07:12

I have blocked bot* using htaccess:

 RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]


I have been informed that this can interfere with traffic, what should I do? should I wait for it to happen again then check logs for IP/Agent name or continue to block unkown robots?

I did a dns lookup on the googlebot entries I do have and they check out.
2:07 am on July 31, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


That's not vey likely to 'interfere with traffic' (whatever that means) because you have specified that the user-agent name must begin with "bot". That is quite unlikely to match any legitimate user-agent -- browser or robot.

You might want to check out our Spider Indentification forum for some ideas after you get this immediate problem taken care of.

BTW, your code is a bit wonky, and might be better written as:

RewriteCond %{HTTP_USER_AGENT} ^bot [NC]
RewriteRule ^ - [F]

which is entirely equivalent, but shorter.

Jim
2:13 am on July 31, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:6964
votes: 385


Also agree to check out the Spider ID forum [webmasterworld.com...]

Boffing ^bot won't do much good and targeting "bot" (no quotes) of any kind will kill the big two and a few other beneficial bots at the same time. Spend some time in the above linked forum for some amazing information.