Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
Key_Master - 10:39 pm on Feb 29, 2012 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4421083.htm
Sorry, I can't reveal more than what I've already shared.
Have you thought about about caching robots.txt or putting some sort of system in place to verify that EdisterBot is hitting the right robots.txt?
Brought to you by WebmasterWorld: http://www.webmasterworld.com