wilderness
msg:4065117 | 4:59 pm on Jan 21, 2010 (gmt 0) |
There's no reason that any webmaster should allow visitors (bots or otherwise) when the UA is blank. deny blank user agent (Webmaster World) [google.com] deny blank user agent (google) [google.com]
|
madmatt69
msg:4065168 | 6:36 pm on Jan 21, 2010 (gmt 0) |
Thanks that definitely helped - and now that I've identified about 400+ ip's with no user agent, I'm wondering if there's a way to bulk check them against a blacklist? I have found a few sites that can check individualy, but not in bulk. One interesting thing - Yahoo Japan's bot has been visiting with no user agent..Almost banned them!
|
jdMorgan
msg:4065176 | 6:47 pm on Jan 21, 2010 (gmt 0) |
Certain caching proxies --such as those in AOL's network-- will make HEAD requests with blank UA's. You'll also see a lot of favicon requests with no UA as well. Yahoo's 'ycar' hosts make requests with no UA, but we haven't figured out what 'ycar' is yet... It's still an open question over in our spider and user-agent ID forum. Jim
|
wilderness
msg:4065328 | 10:50 pm on Jan 21, 2010 (gmt 0) |
| I'm wondering if there's a way to bulk check them against a blacklist? |
|
None that I'm aware of.
|
|
