Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Are these considered as bad bots ?
pkKumar - 4:21 am on Jan 8, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4533935.htm
can these be blocked with robots.txt, whether they will follow robots.txt, if not then you will say block in .htaccess but mine is windows server shared hostng. Site is in asp.net. What's the alternative ?
Brought to you by WebmasterWorld: http://www.webmasterworld.com