Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Are these considered as bad bots ?


blend27 - 11:55 pm on Jan 8, 2013 (gmt 0)


I am in the same boat as dstiles, except I run ColdFusion(for over a decade and a half) on IIS/MSSQL/MySQL, no need for Apache.

I see a lot of Windows hosts offer "Mod-Rewrite for IIS" on IIS 6/7/7.5 on share hosting plans. There are also several other flavors out there.

.htaccess(and other Apache bells and whistles) is a great tool, but I only use it when I want to block something and don't want to hear about it(like specific SE spiders from Asia) in my custom logs. I don't use IIS logs, due to the fact that all data is stored in very well optimized & encrypted Schema, then there are custom very well optimized CF Apps written on top of that. This way I keep all the Data, all I need to do is back up DB and Code and I am a history, with the history ;).

@pkKumar,

Do your self a favor and start analyzing request headers from your visitors(programmatically record them), then compare them and learn. Take notes on different UAs. 99% of the Bad Bots and Scrapers have them a$* sideways/backwards and in a wrong order. I wount go into the details on this, But that should get You started!

^ P.S. It is all incrediBILL's fault, he got me thinking, and I now see the light! :) Easy prey for a bot blocker code logic.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4533935.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com