homepage Welcome to WebmasterWorld Guest from 50.17.27.205
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
Ban bots with no user agent
Is it safe to do so? If so, how
madmatt69




msg:4065062
 3:47 pm on Jan 21, 2010 (gmt 0)

Hi all recently my site stats show a bot that just sucked up over a gig of bandwidth. In my 'awstats' report it just defines it as:
Unknown robot (identified by empty user agent string)

Is it safe to assume that any 'legitimate' bot would have a user agent string?

If so, is there a way (via httpd.conf) to deny access to bots that are missing their user agent string?

Thanks!

 

wilderness




msg:4065117
 4:59 pm on Jan 21, 2010 (gmt 0)

There's no reason that any webmaster should allow visitors (bots or otherwise) when the UA is blank.

deny blank user agent (Webmaster World) [google.com]

deny blank user agent (google) [google.com]

madmatt69




msg:4065168
 6:36 pm on Jan 21, 2010 (gmt 0)

Thanks that definitely helped - and now that I've identified about 400+ ip's with no user agent, I'm wondering if there's a way to bulk check them against a blacklist?

I have found a few sites that can check individualy, but not in bulk.

One interesting thing - Yahoo Japan's bot has been visiting with no user agent..Almost banned them!

jdMorgan




msg:4065176
 6:47 pm on Jan 21, 2010 (gmt 0)

Certain caching proxies --such as those in AOL's network-- will make HEAD requests with blank UA's. You'll also see a lot of favicon requests with no UA as well.

Yahoo's 'ycar' hosts make requests with no UA, but we haven't figured out what 'ycar' is yet... It's still an open question over in our spider and user-agent ID forum.

Jim

wilderness




msg:4065328
 10:50 pm on Jan 21, 2010 (gmt 0)

I'm wondering if there's a way to bulk check them against a blacklist?

None that I'm aware of.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved