Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- AVG Toolbar Glitch May Be Causing Visitor Loss


wilderness - 4:52 pm on May 9, 2008 (gmt 0)


Bill,
I've been getting some of these:

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"

which result in 403's and are immediately followed by a full page request (+images) by another browser, which is consistent with a filter software.

However, the many other assorted requests appearing as well may infact be a scraper of some sort.

There was a time when I might install a software to test a UA, however these days I'm not willing to jump through such hoops for invalid UA's and 403-resolutions ;)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3615360.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com