Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- EdisterBot

Key_Master - 10:39 pm on Feb 29, 2012 (gmt 0)


Sorry, I can't reveal more than what I've already shared.

Have you thought about about caching robots.txt or putting some sort of system in place to verify that EdisterBot is hitting the right robots.txt?

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4421083.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com