- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- Reasons for using Googlebot user agent?
jmccormac - 9:25 am on Mar 20, 2010 (gmt 0)
Scrapers use this in an attempt to bypass controls. Some sites will whitelist the large search engine bots but apply access controls and limits to ordinary browsers.
Thread source: http://www.webmasterworld.com/search_engine_spiders/4100609.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com