Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- And Now Google's Doing It. JS Stats Show GoogleBot
keyplyr - 3:36 am on May 14, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4312058.htm
|but this JS file is disallowed, so there's no way a GBot user-agent should be in my JS Stats. |
I've never understood the robots.txt disallow standard to be interpreted as "don't crawl" but only "don't index."
Brought to you by WebmasterWorld: http://www.webmasterworld.com