Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Should I block bot*

Wondering about unknown robots



2:48 pm on Jul 30, 2010 (gmt 0)

5+ Year Member


Bandwidth on one of our sites was severely messed with on the 28th of this month. The cpanel only tracks daily access logs and didnt archive them(it does now), using aw stats I found our bot traffic to be as follows:

Unknown robot (identified by 'bot*') 91541+417 4.78 GB 28 Jul 2010 - 07:12

I have blocked bot* using htaccess:

 RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]

I have been informed that this can interfere with traffic, what should I do? should I wait for it to happen again then check logs for IP/Agent name or continue to block unkown robots?

I did a dns lookup on the googlebot entries I do have and they check out.


2:07 am on Jul 31, 2010 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

That's not vey likely to 'interfere with traffic' (whatever that means) because you have specified that the user-agent name must begin with "bot". That is quite unlikely to match any legitimate user-agent -- browser or robot.

You might want to check out our Spider Indentification forum for some ideas after you get this immediate problem taken care of.

BTW, your code is a bit wonky, and might be better written as:

RewriteCond %{HTTP_USER_AGENT} ^bot [NC]
RewriteRule ^ - [F]

which is entirely equivalent, but shorter.



2:13 am on Jul 31, 2010 (gmt 0)

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Also agree to check out the Spider ID forum [webmasterworld.com...]

Boffing ^bot won't do much good and targeting "bot" (no quotes) of any kind will kill the big two and a few other beneficial bots at the same time. Spend some time in the above linked forum for some amazing information.

Featured Threads

Hot Threads This Week

Hot Threads This Month