Welcome to WebmasterWorld Guest from 50.19.53.104

Forum Moderators: goodroi

Message Too Old, No Replies

basic robots.txt question

     

kapow

4:37 pm on Aug 1, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I am concerned about the major SEs getting the right information about my website.

All of the promotionional pages explaining my services are in a folder called website_design. My clients websites each have their own folder e.g. left_handed_pencils. I don't want spiders that visit my url to index my clients folders (my clients all have their own url).

I know about using 'Disallow' - is it possible to use 'Allow' or something with that meaning so that I don't have to keep changing the robots.txt file?

agerhart

4:41 pm on Aug 1, 2001 (gmt 0)

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member



you could use an "allow = all"

and then specify which ones you want to disallow

Macguru

4:43 pm on Aug 1, 2001 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I believe that if not disalowed from any folder, robots will by default sneak any where they can. Why do not you put all yours clients folders in a "clients" folder and disalow access to it?

kapow

3:22 pm on Aug 2, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



When I buy a domain name for a new client - the host I use (pair.com) has a form that asks which folder the name should point to. It would now be a lot of work to change them all to a sub folder of a clients folder, plus - I don't know if the host system allows that (I can ask) e.g.

clients/furniture
clients/nasa well not yet :)

It would be easier if I could just use some kind of 'Allow only this folder' instruction in my robots.txt

Macguru

4:31 pm on Aug 2, 2001 (gmt 0)

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I do not know how this hosting company work, and I have never seen the form. Do you act as a reseller?

Generally, 1 robots.txt file is required for each domain at it's root level.

Can you please give us more details about ftp file structure?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month