- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- basic robots.txt question
Macguru - 4:43 pm on Aug 1, 2001 (gmt 0)
I believe that if not disalowed from any folder, robots will by default sneak any where they can. Why do not you put all yours clients folders in a "clients" folder and disalow access to it?
Thread source: http://www.webmasterworld.com/robots_txt/108.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com