Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Blocking Spiders
penders - 8:40 pm on Jul 25, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4343689.htm
4) Block known spider IPs? (Is there a list of known spider IPs?) ... would perhaps need updating like the User Agents.
Brought to you by WebmasterWorld: http://www.webmasterworld.com