- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Blocking Spiders
g1smd - 9:15 pm on Jul 25, 2011 (gmt 0)
It's a multi-faceted approach. Once you have a month of raw site logs you will have details for much more than 90% of the bots that might access your site.
After that, you'll just need a minor tweak to the rules now and again.
Thread source: http://www.webmasterworld.com/robots_txt/4343689.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com