Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Are these considered as bad bots ?


not2easy - 4:34 am on Jan 8, 2013 (gmt 0)


You cannot block robots with robots.txt but you can disallow areas of your site or the whole site for compliant robots only. Usually, compliant robots show a URL where you can read their policies. The robots.txt file does not force robots to obey your instructions. On Windows servers, look into using isapi instead of .htaccess, try a search here or ask your host.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4533935.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com