Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt


Brett_Tabke - 3:15 am on Jul 5, 2001 (gmt 0)


We were meaning pages excluded by Robots.txt 2_much. (not banned really, just blocked from indexing).

On another note, it is interesting to block a nonexistent directory with a robots.txt and watch some spiders try to spider it. You can really spot who isn't playing fair that way.


Thread source:: http://www.webmasterworld.com/robots_txt/156.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com