Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Should I block bot*

Wondering about unknown robots

2:48 pm on Jul 30, 2010 (gmt 0)

New User

5+ Year Member

joined:Nov 11, 2008
posts: 12
votes: 0


Bandwidth on one of our sites was severely messed with on the 28th of this month. The cpanel only tracks daily access logs and didnt archive them(it does now), using aw stats I found our bot traffic to be as follows:

Unknown robot (identified by 'bot*') 91541+417 4.78 GB 28 Jul 2010 - 07:12

I have blocked bot* using htaccess:

 RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]

I have been informed that this can interfere with traffic, what should I do? should I wait for it to happen again then check logs for IP/Agent name or continue to block unkown robots?

I did a dns lookup on the googlebot entries I do have and they check out.
2:07 am on July 31, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

That's not vey likely to 'interfere with traffic' (whatever that means) because you have specified that the user-agent name must begin with "bot". That is quite unlikely to match any legitimate user-agent -- browser or robot.

You might want to check out our Spider Indentification forum for some ideas after you get this immediate problem taken care of.

BTW, your code is a bit wonky, and might be better written as:

RewriteCond %{HTTP_USER_AGENT} ^bot [NC]
RewriteRule ^ - [F]

which is entirely equivalent, but shorter.

2:13 am on July 31, 2010 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
votes: 443

Also agree to check out the Spider ID forum [webmasterworld.com...]

Boffing ^bot won't do much good and targeting "bot" (no quotes) of any kind will kill the big two and a few other beneficial bots at the same time. Spend some time in the above linked forum for some amazing information.

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members