Welcome to WebmasterWorld Guest from 54.226.189.112

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Ban bots with no user agent

Is it safe to do so? If so, how

     

madmatt69

3:47 pm on Jan 21, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi all recently my site stats show a bot that just sucked up over a gig of bandwidth. In my 'awstats' report it just defines it as:
Unknown robot (identified by empty user agent string)

Is it safe to assume that any 'legitimate' bot would have a user agent string?

If so, is there a way (via httpd.conf) to deny access to bots that are missing their user agent string?

Thanks!

wilderness

4:59 pm on Jan 21, 2010 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



There's no reason that any webmaster should allow visitors (bots or otherwise) when the UA is blank.

deny blank user agent (Webmaster World) [google.com]

deny blank user agent (google) [google.com]

madmatt69

6:36 pm on Jan 21, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks that definitely helped - and now that I've identified about 400+ ip's with no user agent, I'm wondering if there's a way to bulk check them against a blacklist?

I have found a few sites that can check individualy, but not in bulk.

One interesting thing - Yahoo Japan's bot has been visiting with no user agent..Almost banned them!

jdMorgan

6:47 pm on Jan 21, 2010 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Certain caching proxies --such as those in AOL's network-- will make HEAD requests with blank UA's. You'll also see a lot of favicon requests with no UA as well.

Yahoo's 'ycar' hosts make requests with no UA, but we haven't figured out what 'ycar' is yet... It's still an open question over in our spider and user-agent ID forum.

Jim

wilderness

10:50 pm on Jan 21, 2010 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I'm wondering if there's a way to bulk check them against a blacklist?

None that I'm aware of.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month