Welcome to WebmasterWorld Guest from 54.146.184.210

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Block Bots With Empty User-Agents?

     
6:26 pm on Feb 16, 2011 (gmt 0)

New User

5+ Year Member

joined:Apr 23, 2010
posts:21
votes: 0


Hi guys,

Please can someone tell me how to block bots with empty user agents with .htaccess.

Thanks so much for any help!

my site has been disabled - and i cannot have it turned back on until i stop the problem i am having with bots using my bandwidth.

Many thanks!
12:40 am on Feb 17, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


There should be many examples of the required lines in the archives.

blank+ua [google.com]
7:42 am on Feb 17, 2011 (gmt 0)

New User

5+ Year Member

joined:Apr 23, 2010
posts:21
votes: 0


thanks wilderness -- but im totally usless with .htaccess files - and whenever i have to extract just a snippet of code from a full file, it never works :-(

What i was really hopeing for was just an example of a htaccess that ONLY blocks empty user agent bots. Then i could see which lines are really needed.

Thanks again to anyone reading who can help.

Regards.
5:33 pm on Feb 17, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


FWIW, denying blank User Agents is one of the simplest and least technical additions that may be added to htaccess.

As Jim explained to you in a 2010 inquiry, reviewing the Forum Library, as well as this previous thread link that I supplied, will provide a solution example for you.
2:11 am on Feb 18, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Add above your existing rewriterules:

RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule ^ - [F]

This will block blank user-agents and those consisting of a single hyphen, which are "blank user-agent spoofers" -- and even worse than actually-blank ones...

If you don't already have any other working rules, then you'll probably need to precede the above rule with either both of these lines or only the second one. I can't tell you, because it depends on your host's default configuration. You will have to test on your server to find out.

Options +FollowSymLinks -MultiViews
RewriteEngine on


Jim
7:13 pm on Feb 20, 2011 (gmt 0)

New User

5+ Year Member

joined:Apr 23, 2010
posts:21
votes: 0


I'm sorry for being a total idiot!

wilderness -- I never even saw the link you posted! Sorry. I thought it was just a sig link. haha. I should read more carefully :)

Jim -- Once again, you are a true life saver! Your posts are always fantastic. Cheers!
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members