Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Hiding robots.txt


BlueSky - 1:12 am on Oct 29, 2003 (gmt 0)


Very cool idea. I've also seen someone check out robots.txt via browser before releasing his misbehaving bots on my site.

Some sites fall prey to constant file pilfering, leeching and unwanted mass downloads.

Huh, only some? Gee, I'd love to be in that other group. lol


Thread source:: http://www.webmasterworld.com/robots_txt/74.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com