Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- EdisterBot


keyplyr - 11:42 pm on Feb 27, 2012 (gmt 0)


I PM'd the info, however there's really no need to "fix" anything. It's no different from thousands of other bots that request robots.txt then disobey the directives. That's why I always add a filter to block new bots until I see how they follow the rules.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4421083.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com