Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt


Dreamquick - 3:54 pm on Apr 2, 2003 (gmt 0)


You can't use the an IP address in robots.txt - you need to know the name the robot will look for when it parses robots.txt, also bear in mind that this file is optional rather than server-enforced so there's nothing there that will stop a bad crawler from crawling what they want - irregardless of whether they read that file or not.

Apart from that it looks okay but you might want to throw it through the Robots.txt Validator [searchengineworld.com] just to make sure.

- Tony


Thread source:: http://www.webmasterworld.com/robots_txt/26.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com