Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Filtering Out Really Hard To Find Bad Bots


lucy24 - 3:59 am on Jan 20, 2013 (gmt 0)


My own IP is usually in the middle of a range that I have blocked from several sites (locking myself out more than once and needing to ftp an edit) sometimes, but it jumps all over

That's where mod_rewrite steps in. (Assuming Apache here.) Most of the time a clean and simple "Deny from..." will do, but sometimes you simply have to look at more than one option: "slam the door on this IP range unless the user-agent is me".

Admittedly this is easier if you've got a browser with some distinguishing features. In other words, not Chrome ;) and definitely not a tablet.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4536448.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com