Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- How important is the Robots.txt file now?


Marshall - 11:28 am on Jan 25, 2002 (gmt 0)


Aside from what Brett said about bandwidth, I don't want my sites archived. Some of this is out of copyright concerns, but most of it has to do with why should they benefit from my work. They're not doing me any favors.

Which leads to the issue about what is the point of having a site Terms Of Use which states "can't download, store, save, redistribute..." if these outfits don't obey it. I didn't ask them to spider my site. Aren't they bound by the Terms Of Use the same as a visitor? As far as I am concerned, they are, whether they visit in person or via spider/robot.


Thread source:: http://www.webmasterworld.com/robots_txt/11.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com