Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- EdisterBot


Seedy - 9:58 pm on Feb 29, 2012 (gmt 0)


user-agent: *
Disallow: /keepout
Disallow: /getlost
Disallow: /OKhere/dontgohere


Thing is, even Googlebot doesn't read that. I found this out tonight when they hit my trap page. I promptly replaced the Googlebot specific directory disallow list.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4421083.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com