I am concerned about the major SEs getting the right information about my website.
All of the promotionional pages explaining my services are in a folder called website_design. My clients websites each have their own folder e.g. left_handed_pencils. I don't want spiders that visit my url to index my clients folders (my clients all have their own url).
I know about using 'Disallow' - is it possible to use 'Allow' or something with that meaning so that I don't have to keep changing the robots.txt file?
When I buy a domain name for a new client - the host I use (pair.com) has a form that asks which folder the name should point to. It would now be a lot of work to change them all to a sub folder of a clients folder, plus - I don't know if the host system allows that (I can ask) e.g.
clients/furniture clients/nasa well not yet :)
It would be easier if I could just use some kind of 'Allow only this folder' instruction in my robots.txt