Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- how to disallow feed in robot.txt readable by all bots
tangor - 7:18 am on Aug 16, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4351875.htm
Correct, the regex $ is not required. Done. Otherwise, it correct. Best method is to disallow ALL BOTS then list which bots ARE ALLOWED, but that put me in the minority (called whitelisting)...
Brought to you by WebmasterWorld: http://www.webmasterworld.com