Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Are these considered as bad bots ?


dstiles - 8:31 pm on Jan 8, 2013 (gmt 0)


I've been using only IIS/ASP hosting for over twelve years and wish I'd never dropped linux to begin with! By the time I realised the mistake I had a good-sized library of ASP code. :(

I get over the problem with a file of ASP code that traps various IPs, UAs and methods with the aid of a MySQL database of banned and blocked IPs (there is a difference); the database also holds many known ranges for broadband ISPs. Newly auto-blocked (previously unencountered - typically 10 to 20 a day) IPs are analyzed by me and the full range added to the database.

The code file is included in every web page - takes a bit of extra time to load and test but not significant (typically 15 to 40 milliseconds), even with the MySQL IP database behind it (I have a timer that monitors this) and logging the result(s) to half a dozen text-based logs.

My method was adopted some years ago because, at the time, ASP had only an expensive third-party isapi application. Now, I would recommend using the htaccess-like isapi module under IIS. Advantages of that include control of image serving, which my method does not easily cover.

keyplr - extend that IP range to 202.85.192.0 - 202.85.207.255


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4533935.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com