homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

Should I block bot*
Wondering about unknown robots

5+ Year Member

Msg#: 4179218 posted 2:48 pm on Jul 30, 2010 (gmt 0)


Bandwidth on one of our sites was severely messed with on the 28th of this month. The cpanel only tracks daily access logs and didnt archive them(it does now), using aw stats I found our bot traffic to be as follows:

Unknown robot (identified by 'bot*') 91541+417 4.78 GB 28 Jul 2010 - 07:12

I have blocked bot* using htaccess:

RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]

I have been informed that this can interfere with traffic, what should I do? should I wait for it to happen again then check logs for IP/Agent name or continue to block unkown robots?

I did a dns lookup on the googlebot entries I do have and they check out.



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4179218 posted 2:07 am on Jul 31, 2010 (gmt 0)

That's not vey likely to 'interfere with traffic' (whatever that means) because you have specified that the user-agent name must begin with "bot". That is quite unlikely to match any legitimate user-agent -- browser or robot.

You might want to check out our Spider Indentification forum for some ideas after you get this immediate problem taken care of.

BTW, your code is a bit wonky, and might be better written as:

RewriteCond %{HTTP_USER_AGENT} ^bot [NC]
RewriteRule ^ - [F]

which is entirely equivalent, but shorter.



WebmasterWorld Senior Member tangor us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

Msg#: 4179218 posted 2:13 am on Jul 31, 2010 (gmt 0)

Also agree to check out the Spider ID forum [webmasterworld.com...]

Boffing ^bot won't do much good and targeting "bot" (no quotes) of any kind will kill the big two and a few other beneficial bots at the same time. Spend some time in the above linked forum for some amazing information.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved