Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- How i can control unidentified bots?


lucy24 - 10:13 am on Mar 30, 2012 (gmt 0)


robots.txt has no effect on bad robots, because they probably don't read it and definitely don't obey it.

Blocking robots by htaccess will not prevent them from trying to get in. You will still see them in your logs. But all they take is a few hundred bytes for a 403, instead of the multiple Ks or MBs they would get if they reached the real page.

I disallowed all bad robots by .htaccess and robots.txt

All of them?! How?


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4437712.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com