- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Really necessary to add to robots.txt
coolshop - 9:06 pm on Jul 4, 2003 (gmt 0)
Is it really necessary to add the two lines to robots.txt like the AdSense FAQ says? As long as you are not explicitly banning their bot, why is this necessary?
Thread source: http://www.webmasterworld.com/robots_txt/92.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com