Welcome to WebmasterWorld Guest from 54.161.228.30

Forum Moderators: goodroi

Message Too Old, No Replies

basic robots.txt question

   
4:37 pm on Aug 1, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I am concerned about the major SEs getting the right information about my website.

All of the promotionional pages explaining my services are in a folder called website_design. My clients websites each have their own folder e.g. left_handed_pencils. I don't want spiders that visit my url to index my clients folders (my clients all have their own url).

I know about using 'Disallow' - is it possible to use 'Allow' or something with that meaning so that I don't have to keep changing the robots.txt file?

4:41 pm on Aug 1, 2001 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



you could use an "allow = all"

and then specify which ones you want to disallow

4:43 pm on Aug 1, 2001 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I believe that if not disalowed from any folder, robots will by default sneak any where they can. Why do not you put all yours clients folders in a "clients" folder and disalow access to it?
3:22 pm on Aug 2, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



When I buy a domain name for a new client - the host I use (pair.com) has a form that asks which folder the name should point to. It would now be a lot of work to change them all to a sub folder of a clients folder, plus - I don't know if the host system allows that (I can ask) e.g.

clients/furniture
clients/nasa well not yet :)

It would be easier if I could just use some kind of 'Allow only this folder' instruction in my robots.txt

4:31 pm on Aug 2, 2001 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I do not know how this hosting company work, and I have never seen the form. Do you act as a reseller?

Generally, 1 robots.txt file is required for each domain at it's root level.

Can you please give us more details about ftp file structure?