- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Do we need robots.txt?
willtell - 9:21 pm on Jun 6, 2002 (gmt 0)
Ciml, could you elaborate on your comment about sending 403's. We are going to stop certain spam sites at the network card level.
Thread source: http://www.webmasterworld.com/robots_txt/112.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com