Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- casper bot search attempts to infect sites


dstiles - 8:55 pm on Jul 11, 2010 (gmt 0)


I return a 403 for any page request that includes the extension (amongst others) .php. There is no way I would give them the folders/files they ask for: I would be creating them at the rate of dozens per day! And to no purpose except to encourage them.

I doubt they actually read the files anyway: they are just POSTing to them. Anyone noticed if they abort the read (206, was it?)? I haven't really checked the site logs, only the security logs which don't record that.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4160991.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com