Welcome to WebmasterWorld Guest from 54.225.20.73

Forum Moderators: goodroi

Message Too Old, No Replies

basic robots.txt question

     
4:37 pm on Aug 1, 2001 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2001
posts:1281
votes: 0


I am concerned about the major SEs getting the right information about my website.

All of the promotionional pages explaining my services are in a folder called website_design. My clients websites each have their own folder e.g. left_handed_pencils. I don't want spiders that visit my url to index my clients folders (my clients all have their own url).

I know about using 'Disallow' - is it possible to use 'Allow' or something with that meaning so that I don't have to keep changing the robots.txt file?

4:41 pm on Aug 1, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member agerhart is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 29, 2001
posts:2945
votes: 0


you could use an "allow = all"

and then specify which ones you want to disallow

4:43 pm on Aug 1, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


I believe that if not disalowed from any folder, robots will by default sneak any where they can. Why do not you put all yours clients folders in a "clients" folder and disalow access to it?
3:22 pm on Aug 2, 2001 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2001
posts:1281
votes: 0


When I buy a domain name for a new client - the host I use (pair.com) has a form that asks which folder the name should point to. It would now be a lot of work to change them all to a sub folder of a clients folder, plus - I don't know if the host system allows that (I can ask) e.g.

clients/furniture
clients/nasa well not yet :)

It would be easier if I could just use some kind of 'Allow only this folder' instruction in my robots.txt

4:31 pm on Aug 2, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member macguru is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 30, 2000
posts:3300
votes: 0


I do not know how this hosting company work, and I have never seen the form. Do you act as a reseller?

Generally, 1 robots.txt file is required for each domain at it's root level.

Can you please give us more details about ftp file structure?

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members