Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Put your robots.txt on a diet

Brett_Tabke - 8:36 pm on Oct 28, 2003 (gmt 0)


One thing a few mentioned here about this great post by JD, is #3.

I have alot of near duplicate lines because there are imposter robots out there that try to make you think they are something else. Foobot, may not be the same as foobots.

Thread source:: http://www.webmasterworld.com/robots_txt/3.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com