Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- ip-web-crawler - stupidity never goes out of fashion
dstiles - 6:36 pm on Apr 10, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4563281.htm
> A. Yes. IP-Web-Crawler always checks the robots.txt file first.
So, a bot checking web sites for illegal content that sites with illegal content can block with a simple entry in robots.txt. Delicious!
Brought to you by WebmasterWorld: http://www.webmasterworld.com