Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- basic robots.txt question


Macguru - 4:43 pm on Aug 1, 2001 (gmt 0)


I believe that if not disalowed from any folder, robots will by default sneak any where they can. Why do not you put all yours clients folders in a "clients" folder and disalow access to it?


Thread source:: http://www.webmasterworld.com/robots_txt/108.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com