- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- In the era of increasing numbers of bad bots is robots.txt irrelevant?
amznVibe - 5:23 pm on Feb 20, 2006 (gmt 0)
robots.txt is very useful for one thing:
trapping bad bots! ;-)
and rewrite crawlthisanddie.htm to your htaccess trap, and poof!
no more bad bots...
Thread source: http://www.webmasterworld.com/robots_txt/871.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com