Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- UAs that asked for robots.txt
incrediBILL - 9:41 am on Jul 16, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4340279.htm
Just ran a report for all UAs that requested robots.txt]
I normally wouldn't post a link but I got a 'post too large' error.
Here it is for your viewing pleasure...
Brought to you by WebmasterWorld: http://www.webmasterworld.com