Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- In the era of increasing numbers of bad bots is robots.txt irrelevant?


bears5122 - 8:40 pm on Feb 20, 2006 (gmt 0)


It'll be interesting to see how this is handled over the years as I too believe that robots.txt is outdated. You wonder if we'll hit a day when bot owners must lobby web hosts to allow access to their server. Kind of like an invitation only policy.

The problem with bad bots is only going to get worse with time.


Thread source:: http://www.webmasterworld.com/robots_txt/871.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com