Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- files requested by robots.txt


rbs10025 - 10:22 pm on Jun 7, 2003 (gmt 0)



DEVIOUS robots in most instances don't even bother with robots.txt. However, there is no need to take a chance on pointing devious bots to either folders or files of which your websites or any other websites DO NOT have links pointing to.

And FWIW, I have on occasion seen entries in my server log indicative of real humans viewing my robots.txt files and then deliverately checking out the "disallowed" directories.


Thread source:: http://www.webmasterworld.com/robots_txt/28.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com