- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Blocking Spiders
penders - 8:40 pm on Jul 25, 2011 (gmt 0)
4) Block known spider IPs? (Is there a list of known spider IPs?) ... would perhaps need updating like the User Agents.
Thread source: http://www.webmasterworld.com/robots_txt/4343689.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com