- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Blocking Spiders
tangor - 8:50 pm on Jul 25, 2011 (gmt 0)
Work smart, not hard... WHITELIST the bots allowed and disallow all the rest. These days that list of who's invited in is MUCH SMALLER than the other way around.
Thread source: http://www.webmasterworld.com/robots_txt/4343689.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com