homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

basic robots.txt question

 4:37 pm on Aug 1, 2001 (gmt 0)

I am concerned about the major SEs getting the right information about my website.

All of the promotionional pages explaining my services are in a folder called website_design. My clients websites each have their own folder e.g. left_handed_pencils. I don't want spiders that visit my url to index my clients folders (my clients all have their own url).

I know about using 'Disallow' - is it possible to use 'Allow' or something with that meaning so that I don't have to keep changing the robots.txt file?



 4:41 pm on Aug 1, 2001 (gmt 0)

you could use an "allow = all"

and then specify which ones you want to disallow


 4:43 pm on Aug 1, 2001 (gmt 0)

I believe that if not disalowed from any folder, robots will by default sneak any where they can. Why do not you put all yours clients folders in a "clients" folder and disalow access to it?


 3:22 pm on Aug 2, 2001 (gmt 0)

When I buy a domain name for a new client - the host I use (pair.com) has a form that asks which folder the name should point to. It would now be a lot of work to change them all to a sub folder of a clients folder, plus - I don't know if the host system allows that (I can ask) e.g.

clients/nasa well not yet :)

It would be easier if I could just use some kind of 'Allow only this folder' instruction in my robots.txt


 4:31 pm on Aug 2, 2001 (gmt 0)

I do not know how this hosting company work, and I have never seen the form. Do you act as a reseller?

Generally, 1 robots.txt file is required for each domain at it's root level.

Can you please give us more details about ftp file structure?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved