Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Stopping scrapers from the get-go


wilderness - 3:20 am on Feb 16, 2011 (gmt 0)


5) block google and the other bots from crawling my images. I think this will block all robots from crawling gif's at any level of my site?
User-agent: Googlebot
Disallow: /*.gif$


There's a more effective way to accomplish this!
Place all your images in specific and multiple image directories, and include these image directories in robots.txt.
The compliant SE's will abide.
Any lack of compliance is easily noticeable and adjustments and notations would then be made on both the IP range, and the UA.

Structure of your website (s) and consistency of methods in structure, will allow you to recognize suspicious visitors whom are traveling your website in a method outside of the built-structure (i. e., crawling).


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4267704.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com